How Spam Filtering Works: From SPF to DKIM to Blacklists
Have you ever wondered why it’s so often recommended to use a dedicated email service like Amazon SES, Mailgun, or other email services? Maybe you’d rather set up your own email server?
I’ve long been guilty of only having a vague understanding of what goes on under the hood of email, and what causes emails to be sent reliably or sent straight into spam. In this week’s article, I’m going to dig into just that to find out what makes a reliable email service.
What Causes Email to Go to Spam?
We’ve all been there before; even a seemingly solid email setup can have emails dumped into your recipient’s spam folder or filtered before it even gets a chance to end up there. This can lead to lost sales, confusion among your websites’ users, and perhaps an overall bad impression of your company.
The truth is that there’s no one thing that can guarantee good deliverability, and a multitude of factors come into play.
A Lack of SPF and DKIM Records
SPF and DKIM records are both things that I’ve always known were important for deliverability, but never bothered to actually look into beyond the fact that I needed them. It turns out they are actually both very simple.
Sender Policy Framework (SPF) records specify the servers that the domain is allowed to send email from. This is the same concept as a return address in old-school mail – if you get a letter from your mother, but the return address is wrong, you may be suspicious that the letter didn’t actually come from your dear mother.
DKIM records offer another way to verify that the email sender is who they say they are. DKIM stands for Domain Keys Identified Mail, and offer another layer of verification. DKIM relies on the email being sent to contain a DKIM signature, which is verified against the DKIM DNS record for that domain.
A private key is used by the email sending service to create a DKIM signature hash email header from one or more important parts of the email that should not change, such as the “From:” header and the message body.
The DNS record acts as a public key – the receiving mailbox provider uses it to decrypt the DKIM hash signature back to the original text and verifies that the message still has that same text. This ensures that the message has been sent from the correct domain, and that the email hasn’t been altered in any way.
Not Using Dedicated IP Addresses(?)
If you research the various email providers, you’ll notice that some of them advertise a dedicated IP address you can use for your email. By sending your emails from one dedicated IP address, you don’t run the risk of sending email from the same IP address that has been (or is still being) used to send emails that are being reported as spam.
Sounds like a no-brainer right? We should all hop on the dedicated IP address train to better email deliverability?
Unfortunately it’s not that simple. If you don’t send enough emails to build up a reputation on your dedicated IP address, spam filters won’t know who you are and are more likely to block you. Also, if you get a fresh IP address and immediately start sending out a huge amount of emails, your sending pattern may be confused with a spammer. Spam filters need to trust you first.
The key to successfully adopting and using a dedicated IP address for emails is to send a consistent amount of emails and keep an eye on your reputation. You should also “break in” the new IP address by sending a smaller amount of emails at first and gradually working up to your full email capacity, otherwise known as “warming up” the dedicated IP address.
If you’re interested in reading more about this, Amazon has a great guide comparing the pros and cons of a dedicated IP address. The guide is written for Amazon SES, but the concepts behind it can apply to any other email service that offers a choice between a shared or dedicated IP address.
Showing Up on Email Blacklists
After all this talk of reputation, I should probably explain what exactly sender reputation is. When you send an email, the recipient’s email service likely checks the domain and IP address that you use to send email and compares them against a blacklist.
There are hundreds of blacklists run by as many different blacklist providers, each containing a database of IP addresses and domain names that are used by spammers. These blacklists mostly rely on users reporting email as spam, so it’s important to monitor your email complaints and make any necessary changes to ensure that you don’t end up on one of these lists. If you do end up on a blacklist, you need to reach out to the blacklist provider for steps to get removed.
A great tool to check to see if you are on any blacklists is MxToolbox. You can provide the domain or IP address you’re using for mail, and within a few seconds it will check dozens of the most popular blacklists.
MxToolbox also offers a service that can automatically monitor your blacklist status. The free version includes weekly automatic checks for one IP address, and scales up from there to include hourly or real-time monitoring for multiple IP addresses or domain names.
Getting Hacked
In this day and age it seems almost impossible not to get hacked. If you’re running your own mail server, it will need constant attention to stay updated and patched against new vulnerabilities.
The problem is compounded when a web server is also acting as a mail server. Neglected content management systems are incredibly easy for a skilled hacker to locate and exploit, at which point it’s possible that they will use the server to send their own mail. And they won’t care much about your reputation!
How Can a Dedicated Email Service Help?
A dedicated email service won’t be the be-all and end-all solution to email deliverability, but it can definitely help make a lot of these things easier.
For example, Amazon SES automatically sets up SPF and DKIM for its own email servers so that if you don’t set them up yourself, your emails will still pass most spam checks. It does this by setting the “From” field to an alias of Amazon SES, so that SPF and DKIM both pass for amazonses.com:
This results in a “Sent via amazonses.com” message in most email clients, but that’s better than failing and ending up in a spam folder. Of course, you can (and should) set up your own SPF and DKIM records. Our (now released) plugin, WP Offload SES will guide you swiftly and easily through this process, step-by-step.
Most dedicated email services will also monitor their own IP addresses in email blacklists, so that’s one less thing that you have to worry about checking yourself. If you run a dedicated IP address, you may have to be more vigilant, but it’s still safer than setting up a dedicated email server of your own.
Conclusion
Sending email, while seemingly simple, has a lot going on behind the scenes and is not something to be taken lightly. While it is technically possible to run email through your own server and do all of these things yourself, it’s not something that any reasonable person would want to keep up with. Using a transactional email service like Amazon SES will save you the time and hassle of worrying about every small detail, and will ultimately be more secure in the long run.
We’re working on something that’s going to make it easier to get yourself set up with a dedicated email service (Amazon SES) and WordPress so you can avoid some of these headaches in the future. (Edit: it’s since been released! Check it out here.)
Have you tried running your own mail server before? What made you give up and use a dedicated email service? Let us know in the comments.
Source: https://deliciousbrains.com/how-spam-filters-works/
