So why am I writing all of this? Unfortunately, email is starting to become synonymous with Google’s mail, and Google’s machines have decided that mail from my server is simply not worth receiving. Being a good administrator and a well-behaved player on the network is no longer enough.
This is already a big philosphical problem now, and it will only get worse as large tech companies try to wrestle ever more control over the web away from users. And because this sort of stuff is so low-level and technical, it’s not going to grab headlines or stirr the masses.
I’m sympathetic, as a former email server admin. But I’m also not sympathetic, because I used to be an email server admin. It sucks running your own server. Spam is virtually impossible to filter out. Users were increasingly upset that spam was getting through, their email was being bounced by someone else because that random email server was terrible.
Gmail isn’t email, but email is best done by a medium to large provider that can specialize in it. Maybe some one will come up with a better protocol thats less possible to abuse, but until then there are a ton of other non google providers of email.
I mean its not like Slack/ facebook message/whatsapp/ instagram/snapchat/twitter where there is only one possible provider.
But yeah, you should not be running your own email server. Thats silly these days.
Bill Shooter of Bul,
I respect your own choices, but IMHO what you are saying goes too far. What is your reasoning that little guys should hand control over their infrastructure to the big guys? There’s both a lot of pros and cons that go into outsourcing everything, but let’s be clear it’s not all pros. In many cases the small businesses I work for do not want to hand functions over to google, amazon, etc since they already have way too much power over local businesses. This is actually a common gripe among my clients. While my professional work may bias my opinion on the subject, I think it’s foolish to leave everything to the big corporations. I think in the future we’ll come to see that letting a few giant corporations control the world was not such a wise thing to do. The success of corporate giants, with their billions/trillions, often comes at the expense of local companies and jobs.
I’m pretty sure that Bill Shooter of Bul was _not_ suggesting out-sourcing to Gmail. There are a range of “little guys” who run competent mail servers that you can out-source to. Also a range of non-Google big guys. My employer recently went through the on-premises Exchange server, off-site (colo) Exchange server to Microsoft’s Outlook service, and I’m sure that pattern is driving Microsoft’s cloud service mightily. There are ISPs that still provide functioning SMTP service, which gets you some measure of “local”. I use my ISP’s mail server for personal SMTP, after long ago abandoning the work of maintaining my own. They’re quite competent (although they did have a worrying phase where it looked as though they too would out-source to Outlook, that has now passed). There are a bunch of specialists like PoBox. The lynch-pin for me was long before the reported issues with Gmail, when the several BSD mailing lists that I subscribe to started dropping personal-hosted mail on the floor.
I do run a personal IMAP server. There’s no way I’ll leave my mail _storage_ up to anyone else.
SMTP is another issue though. Go ahead: authenticate against known infrastructure. Not that hard.
areilly,
I’m just a tad frustrated by that comment because when he says email is best done through a medium to large provider at the exclusion of small providers like myself, it’s a bit difficult not to take that personally. I know he may not mean it that way, but business is hard enough for the little guys even before people start suggesting technology tasks should be left to the bigger companies. We work our butts off and I often feel we go out of our way to serve clients requests when larger companies don’t care. Oh well, part of my frustration could be that he’s not entirely wrong, it just makes me feel unappreciated.
I’m with you on this, not because I host email for others but because I use a small hosting company to serve my email and web hosting needs, and they are absolutely fantastic at both tasks. People often overlook the huge advantage of being served by a company small enough to respond personally to support and feature requests rather than having to wait for a Google or Bluehost automated response (or in the former case, often no response at all, even automated).
The tiny hosting company I use, Tiger Technologies, has been around since 1994 and survived both the late 90s dotcom bubble and the explosion of “me too” reseller hosting of the past decade. They have been rock solid and quick to address any mail or web hosting issues that I’ve had over the years, and I can’t say that for any other service short of rolling my own with Vultr or HostUS VPS service. Those services are great for infrastructure issues, but I don’t have time to be my own sysadmin and I’m happy to let the little guy handle it all for me.
Just remember that once it’s on a provider’s server, it can be obtained after 180 days without a warrant, which does not apply if you run your own server (see https://en.wikipedia.org/wiki/Email_privacy#Federal_statutes .) I’m not defending this, but it is what it is, and it may be a factor to consider.
I’m running my own email server. One VM instance (enterprise linux) filters out spam, not always perfect but good enough for me, and another instance takes care of storage and IMAP. Setting up DMARC and friends takes some time, but there are plenty of tutorials on the webs.
Also I keep an aye on the mailops mailing list and in general I get the impression that the big players really try to find a balance between spam control and interoperability. It just isn’t easy.
As some of the comments also mention on his site: you really must set up DMARC.
In return, you get all the nice stuff… piping mail aliases to scripts, nearly unlimited storage, the ability to do easy data analysis on your own mail, and the fun of having your own toys.
I also run my own mailserver and whilst there’s more hoops now than there were 10 years ago it’s still rewarding.
Make sure you have rdns, SPF, DMARC, dkim and now mta-sts, spam assassin/rspamd and you should be ok.
I don’t think Google or Hotmail/Outlook are trying to break other people’s mail servers, it’s just that they don’t think of them at all when they try to block spam at the scale they operate.
Google is feeling like the Microsoft of the 90s. They have and exert way too much control over the internet and it’s standards now.
It’s a general problem with Google IMO. “We run this service that can be abused, so we have created this automated filter that we won’t tell anyone how it works (or publish any concrete guidelines) so the abusers cannot get around it easily and so we won’t give a free leg up to the competition”. Take your pick: Content ID, Bouncer, Blogspot censorbot, Gmail spam filter. It’s the same modus operandi all over.
And as Google’s services become the default provider for a number of services, Google becomes the default police of the internet. Enforcing rules kept secret.
What’s the solution here? Providing a method of real ID confirmation (like PayPal does for sellers) so Google’s services can’t be anonymously (pseudonymously) abused? Forcing Google to reveal how their automatic filtering works?
kurkosdr,
I agree. I’ve witnessed false positives where google (as well as other major providers) erroneously fail to deliver an email. Most of the time we track these cases down when a google user calls us, but it’s technically google’s fault and the ball is in their court – the problem is they’re not very responsive to their own users. Google services are obviously extremely popular; I use some of their services in the course of my work and it’s usually fine, but I’d seriously have to give them an F grade on support. When things do go wrong, google is totally MIA. If us regular folk offered support comparable to google’s, we’d rightfully be out of business 🙁
Are you using Google services as The Product, or as a (paying) Customer? Just asking…
evert,
I don’t have any google account for me personally (ignoring the account they force you to have in order to activate android devices and use the app store), however I often have to use my client’s accounts in the course of business. Some but not all are paid.
And, if I interpret your comment right, then paying customers also get poor support?
I must say I’m not surprised. Support doesn’t scale, and Google does like stuff that scales.
evert,
Yeah, one client a few months ago got upset over google support and chewed my ear off for it, haha. But google can get away with it. Sometimes I try to encourage clients to go elsewhere but realistically google is at the center and at the edges of the advertising universe. As as website admin I have to accept that. What do you do about it? Shrug.
Starting a mailing list today seems nearly impossible. I attempted to start a well-configured GNU Mailman list last year for my club, but it was simply impossible to get the notifications past Google’s spam filters. This is a serious internet regression and a great loss.
There were lots of discussions on mailing lists and SPF / DMARC. It is indeed very hard to do it right, maybe impossible. At a minimum, you have to rewrite the FROM header. Probably also remove the DKIM signatures and create new ones.
I’m a long time Gmail user that’s looking for less Google in my life. Does anyone have any recommendations for email providers that have solid spam filters but not overly so? I don’t mind paying a small fee. A few people have suggested Proton Mail, but would like to read more opinions.
Thanks
I mentioned above that I use Tiger Technologies (tigertech.net) for email and web hosting, they are great if you have your own domain(s) and are looking to move to a smaller but robust and reliable host. If you just need email and don’t need web hosting, Fastmail is excellent. A few years ago I switched my [email protected] identity to [email protected] so I’d have a secure webmail address that doesn’t depend on my domains, and they are nothing short of phenomenal. Lots of extra perks like file storage and notes/calendar that are (in my experience) far better than self-hosted solutions like Nextcloud. They are based in Australia and they have both personal and business tiers.
So I just checked some of my logs, and this actually explains why I “couldn’t get OpenSMTPD” working for sending email. It worked just fine, my gmail test account was just refusing it.
It’s amusing, since Google has consistently been terrible at taking action against spam FROM GMail accounts. There were already plenty of Indian SEO/web dev spammers using @gmail.com addresses, and in the last year or two I’ve started seeing spammers who appear to be using paid G-Suite accounts (their messages come from non-Google domains, but their MX records point to Google’s servers).
I posted about it on a Google support forum a while back and was told that the process of reporting spam from G-Suite accounts is different, and they can’t simply be reported to the general abuse address, but have to be sent to the abuse or postmaster account under the custom domain that the spam came from. They apparently forgot to mention that to any 3rd-party services like Spamcop, and for some reason stopped responding when I asked for confirmation that the only way to report spam from G-Suite accounts involved potentially doxxing myself to the spammers I’m reporting…