To be clear, you absolutely can still run your own email infrastructure, getting email delivered to you, filtering incoming spam, sending email (with DMARC signatures and other modern email practices), providing IMAP access, and even run your own webmail setup. You can even do this with all open source software. But the email environment you get this way is increasingly what I called an artisanal one. It’s cute, decent enough, and hand-crafted, but it doesn’t measure up in usability, features, and performance to the email infrastructure that is run by big providers. Your IMAP access might be as good as theirs, but things like your webmail, your spam filtering, and almost certainly your general security will not be as good as they have.
In short, if you run your own email infrastructure, it will not be up to the general quality you could get from outsourcing to big providers (they can’t really be called specialists). And you cannot fix this by trying harder, nor with the magical right choice of open source software, nor with the magical right choice of commercial software. Entirely “on premise” email is now an inferior thing for almost everyone.
I’ve always wanted to try and run my own email server, but I’d never run my main email address myself, since my income and interactions with the government depend on it. Still, it’d be a fun side project.
I’m running my personal mailserver and I like it. But I have to agree with the author that for most organisations, running one’s own mailserver doesn’t cut it.
Still there are some uses for one’s own mailserver that are attractive, like:
– just add another drive for whatever amount of mail storage you need
– custom scripts executing when some mail arrives (e.g. using aliases)
– very user-specific spam filtering
– security/privacy… storage on one’s own encrypted drive, no need to trust the anonymous admins of big corps
– ability to run custom scrips / analytics on one’s mailstore
But mostly you do it for the fun of it. Once you’ve set it up correctly, it doesn’t take much parenting. Postfix, dovecot and the likes are very stable, secure, and for your small server they don’t require expensive hardware.
(I also run an Exchange server for fun, a box that needs much more continuous attention and also eats much CPU and RAM for my single user. But calendaring is still much better in the Microsoft universe.)
I use SOGo for my calendaring. Running your own email server is quite nice. I enjoy it a lot.
I’m considering SoGo. I’m not sure though if it supports (binary) attachments to appointments. Using Outlook/Exchange, I use that feature a lot.
Ech, Maybe if you’re doing it for yourself. I could never handle the spam volume on my domain, none of the tools were effective enough. Spam used to be very dumb phrama, nigerian prince, and bad ebay paypal phishing. Now, its beyond my capabilities. I get too much not spam in my spam and too much real difficult to detect phishing in the inbox. I gave up. I’ll pay the pros who have time to focus on the problem.
As for the scripts, what not I had a few of those but I did a lot of it client side which is still possible for me. Pop3/ imap are still supported by a number of providers.
Dunno, I use some DNS blacklists (aggressively) and other tools, very low spam levels, I’ve even discontinued my use of bogofilter. But with bogofilter on your personal mailstore, I did get very good results in the past.
I love server side scripts but sure client side is always possible, although more complicated als less reliable IMHO.
Looks like everyone can pay not only for a dedicated server plus colocation but for a domain name, domain zone hosting, backups and to top it off you also need to pay for extra hard drives! Wow. What a nice idea. And you mustn’t forget to pay for all of that regularly or your entire setup goes bust!
Necessary maybe for 0.001% of people in the world.
From my experience nothing custom beats gmail/live/yahoo and all “custom” open source SPAM filtering solutions terribly suck. Once your email is leaked to spam databases, you’ll be inundated with spam.
ProtonMail
Overall a crazy expensive delicate idea which requires a ton of maintenance with very dubious benefits.
I wrote: “I’m running my personal mailserver and I like it. But I have to agree with the author that for most organisations, running one’s own mailserver doesn’t cut it. […] But mostly you do it for the fun of it. […]”
I’m not sure why you mention ProtonMail in reply to my comment.
Anyway.
Funny enough, it’s the reliability of delivery if I get something wrong that has me constantly procrastinating migrating off GMail.
Webmail? I run Thunderbird and want to develop my own UI unlike anything currently in existence by resurrecting what I started as my final project for my bachelor’s degree.
Anti-spam? Too unreliable. I dream of the day when I don’t have to set a blanket “not spam” filter and use a milter to set up a more automatic version of the “everyone gets their own alias, which I can revoke if they start to spam me” tactic I currently use. (Specifically, I want to be able to have each alias refuse messages not from the person I gave it to)
General security? I’m sick of having to dance around Google’s increasing attempts to try to get me to add an SMS number. (Landline by choice here and Google already knows enough about me.)
Oh, clarification.
I dream of the day when I (don’t have to set a blanket “not spam” filter) and ([when I can] use a milter to set up a more automatic version of the “everyone gets their own alias, which I can revoke if they start to spam me” tactic [which] I currently use).
ssokolow,
Doing it yourself from scratch is a painstaking process with lots of trial and error. But if you’ve got a guide or a little help then you can take a lot of shortcuts to get there quicker. Unfortunately though even if everything is correct on your end, it doesn’t automatically mean everything going to work flawlessly because sometimes providers will block your emails on their end and there’s nothing you can do about it short of getting them to fix it. I would not say it’s a frequent problem, but something to be aware of so that you’re not under false expectations that your server will be treated fairly.
We need more competition in this space, alas I don’t think there’s a business model that makes it work. No matter how good your software is, there’s no way to beat the giants 🙁
Yes, IMHO everyone should do this. It’s because I used a variant email for osnews that I am able to detect when the osnews database got leaked a few years back. Ideally we’d have a better email where one would need a valid signature in order to use emails, it would be the end of spam forever. But it would require a trivial means of exchanging email keys today that we don’t have in any universal sense today. And while it’s easy to build the technology to do this, it’s next to impossible to convince the world to use it.
You too huh? I agree they’re prying too far into our lives and devices. Unfortunately the network effects are making it increasingly difficult to opt out of anything. Even kids can be forced to have google accounts through school. Even though there there are laws protecting kids, I’m still not happy about it.
Thom Holwerda,
SMTP requires so many layers of hacks that I for one would like to see it dumped and replaced….however realistically if that happened I’d worry that some of the best open & federated networking aspects that have been grandfathered into our email standards would get sabotaged by the corporate giants who don’t give a crap about network neutrality.
I find the real-time blacklisting works fairly well for self hosted email with a major caveat, ironically the real time blacklists are helpless against mail originating from google’s gmail servers. They cannot get blacklisted because they’re “too big to fail”. So even though the majority of spam I get (after blacklisting) can be tracked back to google’s own severs, their servers cannot be blocked because so many rely on gmail. Gmail is fast to act on inbound spam, but outbound spam is another story. Just trying to contact google abuse as an external operator is futile & miserable. They’re so much worse than other providers, I simply don’t have these problems with other providers like microsoft, yahoo, proton mail, etc who by contrast actually have responsive admins.
When the shoe is on the other foot and your sending email to gmail recipients, I’ve never had any issues with gmail. However I have had issues with some other operators, the worse of which will accept email successfully yet silently drop them without a bounce so there’s no record or evidence of emails being dropped at all. Such cases are extremely frustrating because the first you learn of a problem is when a user calls, and then you get the blame even if the big operator is technically at fault and logs prove you delivered the mail to them. Try to explain that to a user who reasonably wants their emails to get through.
It’s one of those catch-22s, the more problems big providers cause for small providers (whether intentional or not), the more incentive customers have to switch to big providers. Arguably google has a rather large incentive to keep the outbound spam spigot open to encourage more businesses/users to switch to gmail.
I took a closer look at one of these google originated spams, it passes google’s SPF and DKIM signature checks, just like any legitimate mail would. However I noticed something just now, they contain a line “gmailapi.google.com with HTTPREST”.
(I cut out most of the lines here, but for example…)
It makes a lot of sense that a spammer would use gmail API to send spam. Given that google reports it’s use, this makes me wonder if it might be a good marker to help flag spam coming from gmail. I suspect probably 100% of the spams will have this gmailapi marker, the big question is false positives and how many legit emails contain this marker? I haven’t found any that do yet. Maybe I’ll try to research this further.
Alfman,
Email being a very open platform means it will have issues in the modern age. Like the Telnet of old times, assuming everyone playing nice is no longer feasible.
Anyone can:
And deliver a mail from Bezos to Gates (my dialect is probably off). Now, it will go to a spam folder, and your IP will probably be put on a black list somewhere. But the basic protocol has not changed over time.
On gmail, yes I still deliver my server alerts directly to my personal account. For some reason they don’t get blocked, and I am happy.
sukru,
Well, strong security and crypto are available with SMTP extensions, however these extensions are optional and implementations are far from consistent such that we can’t really count on anything working all the time. This is why being an email admin is so convoluted. If you harden your server, you can end up with users complaining about legitimate emails being blocked. I had this happen with local government emails being blocked and when I looked into it sure enough they were right: The county government’s email system had improperly configured DNS/SPF the result being official government emails being bounced. I try to walk them through steps needed to fix their system (https://mxtoolbox.com/ is a great resource for this BTW), but sometimes they don’t care and make it our responsibility to whitelist them rather than their responsibility to fix their configuration. Ultimately when push comes to shove they often have the ability to call the shots. Regressive though it may be, the little guys don’t carry much weight.
I agree, I can’t remember an instance of delivery problems with gmail. To be clear though my earlier gripe wasn’t with failed delivery but rather the spam originating from google’s network to external addresses. IMHO google’s been awful about addressing gmail’s role in delivering spam to external servers. As a google employee, I wonder if you could convince them to finally fix that because they won’t listen to me or my abuse reports 🙁
Alfman,
Unfortunately I don’t have knowledge on how gmail teams would operate. One certain thing is, they would not want fake throwaway accounts that generate spam. It costs money and good will. But for the rest we can only speculate.
If we were to speculate, I could design a system checking incoming spam at email level, but outgoing ones would be per account.
Because, even if the received email is from a known sender, they could be compromised. So, each individual email content needs to be checked against spam filter.
But, if my internal user is trusted, I would not want to have a false positive on a legitimate mass mail thread. They could be a teacher, for example.
(Of course viruses should be checked all times).
This is just a thought experiment on my end. But would fit the behavior you’d observed.
I’ve been running my own mail server since 2003. It’s gotten to be more and more of a pain every year. You can block a lot of spam with something like rspamd. It’s much better than the old days with spam assassin. However, it’s rather aggressive and messages get dropped occasionally. Many sites that send email now such as banks, universal rewards, etc are coded poorly and think a single bounce means your email address is invalid. They falsely flag your mail as down because it was greylisted or because you were rebooting your server just then. Even if you have a secondary mx, they fail to deliver the mail properly.
Another issue is that spammers tend to target a secondary mx hard. If you set one up, it’s going to get a lot of spam traffic hitting it. I run my primary on a business cable package and have a secondary with ovh. It’s been quite the mess on that ovh server.
I’ve had issues with gmail receiving and sending to me on a few occasions but they’re not as bad as yahoo mail was. At least once a year I’ll get on a spam list and have to fight that. in my case, I briefly tried gmail for a few days but went back to my own server. The issue has been mailing lists for my bsd project. Google’s list solution sucks and can’t be migrated to without issues.
As for webmail, i don’t like it but my wife does. I’ve setup roundcube and it’s good enough for her. She still checks her mail on her iphone most of the time anyway using apple mail.
With per user charging for domains at many providers, it’s much cheaper to provide addresses to project contributors via my own server too.
I’ve given up on the secondary MX for my personal server. Although I’ve consideren proxying the SMTP traffic from secondary to primary while the primary is up; or just disabling the smtpd on secondary as long as secondary can ping primary.
https://blog.zensoftware.co.uk/2012/07/02/why-we-tend-to-recommend-not-having-a-secondary-mx-these-days/
Everyone is chasing the tech arguments. The main problem is legislative.
Email is infrastructure. The basic concepts already exist within telecoms security and privacy and environmental and fraud legislation. Not to put too fine a point on it but every real world physical address and identities organisations and people as well as traditional postal and telecoms systems are pretty much known as are the costs. It’s a minor legislative change or perhaps only a change in statutory guidelines to extend this to email. Problem solved.
Yes you could still have “pop up” providers and false identities and exploitation of weaknesses in countries which have less than adequate responses but almost all of this would fit under criminal legislation and attract jail sentences and significant fines.
The reason why nothing is happening on this front is too many nation states are up to tricks and organisations are either not actively lobbying politicians who, let’s face it, are not technical people nor should you expect them to be, or dipping their fingers in the data stream or profiting from the tawdry state of affairs by selling their own infrastructure solutions to infrastructure problems they caused.
By focusing on the technical issues you’re identifying the problems but getting caught in a repeating loop of inadequate technical responses. The problem is not technical. It’s public policy. And from what I can gather I am the only person on this forum who has ever actively lobbied or sat in the room with politicians. I actually have got action on issues I’ve raised in the past including but not limited to diplomatic human rights action, changes in planning regulation, regulatory action on inadequate and negligent provision of services. Politicians or political party officials have also followed my advice within their personal lives. One now has the job I recommended to them. Another, a QC, took leave from politics to recover their mental health. A very significant case pending within the UK legal system happened in large part because of my lobbying both on points of law and within earshot of a NGO with an interest in the matter and the help of a politically active person with a public profile.
The lesson is we can sit around in a holy huddle discussing the technical issues for ever and a day and blowing off with generic cut and paste political statements (which the business as usual types are very happy for you to keep doing) or you can push the system where it matters. They do.
Last thing: with politics things can happen very quickly but for anything which matters do not expect a result within 1-3 years. It can take a case a year to get to court given the waiting schedules and all the preparation workup front. When it hits government level it can take three years minimum to get one change. Three years is actually pretty good. It can take 7-10, or even 20+ years to get major legislative change.
Start now.
HollyB,
…this isn’t new, we’ve had these debates over decades now. How do you plan on achieving worldwide consensus? In reality, politics never reach the ideals.
It doesn’t seem like you have experience managing SMTP services but there are a lot of problems that stem from very technical issues. Furthermore the reason technical issues exist isn’t because they cannot technically be fixed, but because it would break backwards compatibility and/or result in network fragmentation. Replacing the collective global email infrastructure is an enormous undertaking that many aren’t willing to do in the first place.
I think it can be insightful look at is the IPv6 migration, which has had multiple government mandates going back to 2005, all failed, and a recent report suggests a transition isn’t going to work as long as services remain accessible on the IPv4 network. There’s actually a new mandate under Joe Biden to look at starting IPv6-only services without a transition. Can it finally work this time? Nobody knows. Pragmatic constraints may force this mandate to fail like the rest and for all we know everything he does may be scrapped by the next whitehouse anyways.
https://www.6connect.com/blog/ipv6-government-mandate-what-it-means-for-you/
And your comment on legislation and email is? Devil’s advocate is bad enough but contrarianism and reframing is another level of annoying. You want to do something about the issues or you don’t.
One last thing not all elected representatives get and those who do may well be liars with a vested interest themselves: managers and admin for state agencies often lie through their teeth when they don’t understand the issues or cannot be bothered even where there are statutory obligations. Go hunting for or keep your eye open for compliance and other audits. You’ll often find proof they lied and/or unlawfully underperformed in those.
Go to it tiger, or enjoy sitting in a crappy mess!
P.S. I couldn’t give a damn about the US.
HollyB,
Playing devil’s advocate is a valid way to bring up counter points, however I should note that I was not playing devil’s advocate, everything I said was my real opinion. If consensus were easy, we wouldn’t need legislation to achieve it. Yet because consensus is difficult, legislation becomes difficult as well. You ignored the point about IPv6, but I think it’s a good case study for this. Another example is net-neutrality from a few years back, it was a very popular grass roots movement with the public supporting it to combat abuse, yet almost as soon as it went into effect it got repealed after the political tide changed and republicans took control. Their belief is that corporations should self-regulate.
https://www.theverge.com/2019/7/9/20687903/net-neutrality-was-repealed-a-year-ago-whats-happened-since
IMHO “honesty” can be a political liability and liars can get the upper hand in politics. A political system with no accountability becomes a breeding ground for lies. I’d be the first to argue the importance of fixing this, but there’s a big gap between saying it and it actually happening. Corruption does this.
You’ve got it wrong, I don’t enjoy the mess and I’m not the one holding email back. However I am a realist. I think our email standards will always remain a mess, but the industry may slowly become more consistent with best practices as old systems naturally get replaced over time.
Be that as it may, the US still has a large part to play in global internet standards. You may not give a damn about the US, but I think you’ll have to admit that the US has a lot of weight on issues that you care about.
No sorry Alfman. Playing Devils advocate is a way of being annoying. By taking that position you’re making vast assumptions about other people’s, knowledge, skill, and intent. You’re hijacking the relationship aspect of discussion without consent.
There’s no point in throwing tidbits of hard won experience in your direction if you’re just going to ride roughshod over it like we’re having some kind of verbal pig pong. We’re not.
Yes corruption is a problem and one I am aware of. It’s a much bigger problem in the UK than anyone will admit. When I first called this over a decade ago I go the “If you don’t like it leave” attitude of some people. Then a few years ago an international comparison ranked the UK in an unflattering way. Corruption under the current government is appalling. The kind of corruption which exists in the UK is more tin-pot god careerists taking shortcuts and developing wilful blindness, and empire building, and musing budgets to plug holes in X when they should be spent on Y. Every trick in the book is used and far too many to list. The system works on the surface until it doesn’t then you discover how reckless and negligent and rigged the system can be and how little to no effective remedy exists. The UK would not survive any form of compliance audit.
In fact I told one regulator they didn’t have a clue and the wool was being pulled over their eyes. I explained that advance notice gave staff a warning and put them on their best behaviour. They also did not know what to look for. They had no domain knowledge, they lacked the big picture, they would miss details. In no way were they a safeguard against bad practice. Of course nobody admits or prioritises this and it disappears down a black hole. Now, I keep a very very close eye on public information and am continuously scanning it for “actionable” items. It slipped out in one of the major newspapers the regulator admitted pretty much everything I said. Not only that but their budget was being cut…
No regulator or enforcement authority or decision panel advertises anything other than what they want you to hear. They will talk up specialist units and publish their policy which makes various claims about expertise and training and the process of dealing with an issue. What they don’t quantify is the quality of that expertise or the workload or outcomes satisfaction. As budgets get cut all the pressures heap up and expertise is diluted.
The usual suspects always want a budget increase to maintain headcount or build empires and, very occasionally, spend on something which really matters. Too much of this is often “seat filler” encrusted on authorities and procedures like barnacles. None of them have an interest in getting it right first time and stopping problems being a problem before they are a problem.
Complacency, cruelty, and penny pinching are hallmarks of the UK system.
In my opinion all of the above is deliberate.
I think some EU member states and the EU Commission have got wise to this at least in part.
HollyB,
Let me play devil’s advocate: Not everyone who play’s devil’s advocate is doing it to be annoying, sometimes there’s merit in challenging an unchallenged view.
See what I did there? It can be annoying and be a valid point simultaneously 🙂
Also, this is a public forum where people don’t need “consent” to challenge opinions. Legislation can be part of a solution but IMHO it’s not a sure fix and I think I’ve given good examples of how legislation can fail despite good intentions.
Do you see that these same arguments apply to governments around the world? Governments are anything but ideal and this is why legislative solutions are anything but strait-forward. I simply don’t have that much confidence in legislation fixing email. The easiest technical way to clean up email protocols are to replace the very legacy baggage that everyone depends on. It would create tons of backlash and fragmentation.
Even the big guys do a poor job of running e-mail. E-mail delivery and spam filtering issues is why I would never migrate to GMail.
Setting up and maintaining an email server isn’t exactly easy but it is definitely within a reach of a reasonably advanced Unix user. This is still the best option for organisations that even remotely value their data.
The problem is, what about all those individuals who can’t do it on their own? Aren’t they entitled to privacy?
Email is also the only popular federated communication protocol and this fact is imo worth celebrating.
Some practical observations:
– Spam is overrated. Set up your server correctly, use per-service aliases, don’t use easily guessable addresses like [email protected] and you will be mostly OK. I didn’t even bother setting up a spam filter. I would even say spam is more of an excuse for centralization of email services than a user problem.
– Some big operators (ehm, Microsoft, ehm) are particularly nasty when dealing with, requiring people to apply for a permission to send emails to their domains. They can and do occasionally deny such permissions without an explanation. Imagine if everyone was doing that.
– Use all technical means of authenticating your server to others. There are quite a few of them but nothing particularly difficult to setup. Some (DMARC) can be quite annoying, though, especially when posting to mailing lists.
ndrw,
These are valid points. Inhousing has become marginalized over the past couple decades but I think companies have gotten too cavalier about handing their critical data and business functions to cloud providers. As a result the internet has become very centralized and I think there’s a danger of putting too many eggs in too few baskets.
I think spam is a serious problem, I concede that sometimes this is due to user irresponsibility, but unfortunately data breaches have become routine and their data is being sold even when users are not at fault. I recommend this site for anyone who wants to monitor spamlists for harvasted email addresses. If you have your own domain you can register and get notified of all addresses leaks on your domain!
http://www.haveibeenpwned.com
For example, about 3 years ago osnews was hacked and all our registered email addresses made it into this spam list:
http://www.haveibeenpwned.com/PwnedWebsites#Collection1
I still get spams to my old osnews email to this day.
Additionally, thanks to the increasing hashing power it’s becoming more viable to attack weak email hashes including those used bygravatar. Ihaveibeenpwned has confirmed that some of my email addresses have been harvested in the gravatar attack.
http://www.haveibeenpwned.com/PwnedWebsites#Gravatar
The gravatar weaknesses were discusses a few years ago. I succeeded in reversing email addresses for several osnews members in less than an hour using hashcat’s GPU acceleration. Despite gravatar’s known weaknesses though, many sites including osnews still use it.
http://www.osnews.com/story/128924/what-happened-here/
I get a lot of spam at an email address I used to apply for jobs many years ago. I have many more examples, but you get the idea. These data leaks are a part of life online.
I fully agree. Federated protocols are the only way that we can keep independent services viable.
Yeah, unfortunately DMARC can break legitimate SMTP emails, but for better or worse those are considered casualties of the war on spam. That’s just where we’re at. Personally I wish we could replace SMTP with a clean, secure, universal standard. However I think there’s a huge risk that tech giants would exploit any transition for their own gain by ripping out federated capabilities and forcing everyone to use their centralized services, which would be a terrible loss for the open/federated internet.