On October 3, 2023, Google and Yahoo announced upcoming email security standards to prevent spam, phishing and malware attempts. Outlook.com (formerly Hotmail) is also enforcing these policies.
With the big 3 Email Service Providers (ESP) in agreement, expect widespread adoption soon. Today’s threats are more complex than ever and more ESPs will begin tightening the reigns. Failure to comply with these guidelines will result in emails being blocked beginning April 2024.
In this article, we’re going to cover these guidelines and explain what senders must do in order to achieve and maintain compliance.
↫ XOMedia
Some of these changes – most of them impact bulk senders and spammers – should’ve been implemented ages ago, but seeing them being pushed by the three major email providers, who all happened to be owned, of course, by massive corporations, does raise quite a few red flags. Instinctively, this makes me worried about ulterior motives, especially since running your own email server is already fraught with issues due to the nebulous ways Gmail treats emails coming from small servers.
With the rising interest in self-hosting and things like Mastodon, I hope we’re also going to see a resurgence in hosting your own e-mail. I really don’t like that all my email is going through Gmail – it’s what OSNews uses – but I don’t feel like dealing with all the delivery issues people who try self-hosting email lament about. With a possible renewed wave of interest in it, we might be able to make the process easier and more reliable.
I finally fled Google GMail for ProtonMail. I feel much better now.
ProtonMail walked me through all the DNS SPF/DKIM/DMARC/MX configuration mentioned in the article required to transfer my domains. It only took 10 minutes, and I haven’t had any problems with mail delivery. These new spam technologies are GOOD for small e-mail providers.
I’d been running my own email server since the early ’90s. For a long time, it was a physical server sitting in my office. I relocated, switched ISPs, lost my static IP address, and got tired of jumping through hoops to support a shifting DHCP address. So I switched to a cloud server for a few years. Then I got tired of Gmail and others blocking my email and shifted my domain to Proton a bit over a year ago. I would still prefer to run my own server but it’s just too much hassle any more.
Same. I’ve been running my own mail server for decades. I came the the realization a day or two that i’m getting older, and might as well just pay proton for their premium offering to have one less thing to worry about. Tired of training spamassassin, managing procmail filters, and dealing with random mail servers rejecting my cloud server’s ip ranges.
I do wish proton supported more than one domain on the paid plan, however email forwarding is a thing at most registrars if you’re willing to consolidate email addresses.
I also wish they supported imap/smtp instead of requiring their weird bridge.
Interesting. I fled self hosting over the Spam issue. SPF DKIM DMARC isn’t that bad. Its more the receiving of email that has been the bane of my existence.
Bill Shooter of Bul,
Realtime DNS blacklists are extremely useful here. If I weren’t using them, then I agree with you I’d immediately be swamped with spam. This blacklist lookup tool has been immensely useful in this regard. Plug in a spammer’s IP and you can see all the blacklisted that would have stopped the spam successfully.
https://mxtoolbox.com/blacklists.aspx
You should research the policies behind each blacklist before using them, but these are so valuable for self-hosting! I make sure the rejection returns the blacklist that caused it so that the senders know exactly what’s going on and it helps put pressure on providers to weed out spammers. Win/win.
RBLs are touchy. They’re super useful, but they’ll cut just as quick as they defend. Polluted IP spaces are the top problem with self-hosting, and one of the hardest to workout. SPF, DKIM, DMARC, DNSSEC isn’t hard, but getting clean IP addresses and keeping them clean is the hardest part.f
It’s too bad we don’t have to register an IP block on out own like we have to with domain names. (This is a stream of consciousness thought, and I have no idea how this would work in reality.)
Flatland_Spider,
Yeah, the apews blacklist is an example of an operator with a bad blacklist policy that causes long term damage to IP reputation even after it’s been cleaned up or transferred to a new (innocent) owner. Bad policies are responsible for a lot of collateral damage and should be explicitly avoided for that reason.
This is why we need to look at blacklisting policies before using them. A responsible blacklist operator blocks spam fast and is very effective. But responsible blacklist operators also must offer a viable path for cleaning up IP reputation once the problem is fixed, otherwise it ends up punishing innocent users. Advanced setups like spam assassin can combine blacklists to create a score that’s easy to filter on.
Ideally every single email would be validated by DKIM. A few years ago this was wishful thinking, we couldn’t rely on DKIM being present, but this might actually happen now if the major providers make it mandatory. With universal DKIM, the sender can be cryptographically tracked all the way back to the originating domain!! This is great news all around because IPs are only a roundabout way of representing the sender. It was never a great solution, but in the future we may finally have an effective means of blacklisting just the domains that are responsible for spamming without collateral damage to others who are not.
Another consequence of using IP reputation was that this severely held back the use of IPv6 for SMTP networks. The IPv6 search space is so large that blocking them is hopeless. Spammers could change their IPs every few minutes and never run out. By using DKIM and switching to domain reputation, it will finally allow email providers to move to IPv6 networks! Obviously this won’t happen over night, but I consider it a huge step forward.
Alfman,
I understand why some RBLs list IP subnets instead of individual IPs. The host is less likely to ignore a bunch of people who get their traffic null routed, and I don’t necessarily disagree with the policy.
I’m not sure how many RBLs ask for money to delist IPs. That always seemed shady, and I remember that being rather controversial at the time (like a decade ago).
Yeah, they have been the bane of my existence in the past. Getting put on them is very easy, getting off very difficult. Especially if you are hosting a service for others. But if its for email you yourself are sending and maybe a few other trusted people that don’t have lists they spam with copies of their newsletter, should be fine provided you get a good ip. Which sounds difficult these days.
Flatland_Spider,
I don’t think it’s justified at least in the absence of a larger trend. Obviously they can block entire subnets but to be perfectly honest it is not helpful to end users to block IPs that haven’t been sending spam. Especially when you consider adjacent IPs could belong to innocent companies. For that reason I personally shun the use of intentional collateral damage. Of course when it comes to IPv6, it’s virtually impossible not to use massive subnets because the search space is insanely large.
I personally avoid those. IP addresses are ephemeral for fly by night operators, this means the wrong parties will typically have to face the punitive damages and be on the hook to pay to delist because the spammers are long gone. IMHO such policies will not yield good results.
My hope is that the industry moves to domain based blacklists (thanks to DKIM) and companies can be more directly shunned when their domains start showing up on spammer lists as opposed to generic IPs.
LOL Yes! When I was running an email server for a company, I frequently had to roll back spam protections because other company admins couldn’t setup their servers correctly and would get caught in the nets.
Like Sartre says, “Hell is other people.”
Flatland_Spider,
I’ve encountered those too, email admins who refused to implement best practices and therefor broke your attempts to validate their emails correctly.
I’ve found the easiest way to deal with them is just to whitelist their non-conforming servers while keeping the validation in place for everyone else. My words “tell your provider to fix your end” used to carry no weight. However, going forward, with google, microsoft, yahoo announcing that they will be blocking their emails, non-conforming servers might finally be a thing of the past! I don’t want to count chickens before they hatch, but I think there’s a good reason to be optimistic.
Unfortunately, random unsolicited emails was a source of business for the company, and it was my problem to fix.
Flatland_Spider,
Same. But problems are quite rare and I expect users of email services that don’t comply with basic RFCs would already be accustomed to bounced emails. IMHO this is exactly as it should be, the more bounces they’re getting from everyone the better 🙂
Alfman,
I was more then happy to let them sink, but the partners felt 1 bounced email was too many.
There were a lot of problems, and it was probably related to the companies being ONG companies. This was 10 years ago too, so I’m sure things have changed a lot.
Companies and people migrating to Office 365 and letting MS run the Exchange servers has probably helped quite a bit.
Alfman,
My other complaint from the era was how bad federation between communications tools are. We complain about having to have multiple apps, but email is the one universal tool for companies to talk with each other.
I would can understand admins needing to sign off on the federation and assign access, but that’s not even an option.
Flatland_Spider,
Technology has advanced and we have a much better understanding of the challenges and threat models we need to mitigate. Pretty much everyone agrees that poor standards and hacks make administration harder, features like encryption are very fragmented, less reliable, result in less accountability and more spam, and so on. We really would have the means to create more secure & robust federated networks today…far better than when SMTP was first created and without the absolute mess of add-on hacks over the years.
What is holding back federated networks today has less to do with technological barriers and more to do with rent seeking business models. Our major tech companies would rather push us into proprietary walled gardens than create the next generation federated networks that we should have by now. The case for federated networks is still strong, but unfortunately today’s corporations aren’t that interested in building them any more since their attention has been drawn to more profitable endeavors like user tracking and pushing more ads.
For this reason, we’re probably stuck with SMTP plus hacks for the long term.
Related to this, if someone wants to experiment installing his own mail server and learn how all the pieces fit together, the explanations in https://workaround.org/ispmail-bookworm/ tutorial are wonderful.
I’m running my own Postfix server for receiving mail, and using “smtp2go.com” to handle sending, instead of trying to deal with all the DKIM/PTR/etc stuff myself. Currently handles my personal email, but planning to move my business mail over from Google soon.
Cody Evans,
I use postfix as well. The DNS stuff isn’t too bad actually. It’s pretty easy to test using online resources as well. SPF is more difficult because of the configuration changes required in the daemon. If there’s interest, maybe we could create an osnews article about setting this up?
At one point, Michael W Lucas was working on a book at self-hosting email.
Thom Holwerda,
Yeah. The problem is it works until it doesn’t. I haven’t experienced email delivery problems with outlook. Google has been problematic at times. I’m not currently experiencing any issues now, but when they do reject emails google don’t give a crap about you. The rejection provides a generic FAQ links to best practices that are already being followed. During the same period guess where the majority of our spam was coming from? Gmail! When spam comes from google’s servers, they can’t easily be blocked for spamming because it’s gmail 🙁 It’s not fair but that’s just the reality of dealing with a monopoly.
Protonmail has fans, and they are probably popular enough to bite google back if they tried to pull crap. Maybe osnews could consider using them?
I have been using mailbox.org using my own domain which works quite well. Great service and options and I get to keep my email address even if I want to change provider. Also it is not encrypted contrary to proton meaning I can use any email client I want and also I can actually search my emails easily.
The longest part was to update my email adress on all websites and if possible delete the older accounts.
thomas,
Good, I’m glad you are supporting an alternative
Personally I intend to continue self hosting, I am determined that way. Everything in this article has already been implemented for years. But maybe there’s a chance to convince osnews to switch? Their email is hosted by google, their website is hosted by a google service reseller, their ads are from google too. It’s not ideal to preach alternatives on the one hand while propping up the monopoly on the other. Thom is aware of this, but has stated that it isn’t his role at osnews to make decisions on the technical or business side of things. Obviously I’d like for osnews to change providers, but I feel wary of acting like an entitled prick about it, haha. If they decide to use alternatives, that’d be fantastic, otherwise frankly it’s no different than the majority of my clients who are also paying the google monopoly.
I have used mailbox.org for several years now. It’s great that you are an actual customer to them and not a product to be sold to advertisers.
I switched to Fastmail nearly a decade ago and I love it. It’s not that expensive and the added features are nice. I use the Notes portion of the app all the time, and the file storage is great for accessing the files I use all the time without being dependent on one device manufacturer’s cloud storage. They are quick to respond to issues, very proactive about spam prevention, and they are based outside the US (Australia specifically) for those who may care about that kind of thing.
I’ve tried self hosting in the past and I ran into all the same issues as everyone else, namely being immediately blacklisted whether I’m running from my home network or a colocated server. It seems all the major VM and colo providers are all permanently blacklisted due to abuse, and that means unless I want to pay thousands of dollars a month for a backbone connection I’ll have to quit my job and work full time as my own sysadmin playing whack-a-mole with all the blacklist operators and the big email providers. No thanks.
Polluted address spaces is the problem with self-hosting email, and it’s up to the hosting provider to care. Most don’t, so email servers are limited to the those able to control their own address space.
Flatland_Spider,
Maybe I’ve just been lucky, but I haven’t had problems with polluted address spaces in years. Just last month I provisioned a new /28 network and the IPs came up “clean” in public blacklists.
Some hosts are better then others from what I’ve heard, and some probably see getting blocked as a perk. For instance, Digital Ocean is known to be have bad IP reputations, and Hetzner is known to have good reputations.
That’s a really comprehensive article.
The most interesting part:
I wonder if I can report enough marketing emails to make a difference in their spam score. 😀
Meanwhile I’m getting daily spam emails in Chinese, all originating from hundreds of Outlook and Gmail addresses. (I already created a filter that sends any email containing the 20 most common Chinese characters straight to the trash, but there’s still some emails coming through.)
If it’s a personal mailserver and none of your users understand chinese, there are some options in spamassassin to automatically junk any mail in languages you don’t understand.
That’s an advantage of running your own personal mailserver, a mass market or corporate server doesn’t have the luxury of being able to reject anything in a foreign language, but if you’re the only user and you can only understand one or two languages it makes no sense to allow anything else.
I use a small mailservice (mxroute.com) that I like and trust. Self-sending is possible, but just a headache, even if you jump through all the hoops. But what I discovered was that it’s so easy to receive all your own mail (I use postfix)! I want to receive ALL mail, and then I’ll deal with spam myself. I’ve missed important mail in the past because “it came from China” — yes, that was mail from a bank that I needed to receive… My huge emailarchive I host myself on IMAP, and that works great, access to everything everywhere though the years.
I use MX Route for personal email, and it works well.
MX Route using CPanel to adming the account is really annoying though.