Good morning,
Today is monday the 22nd of January 2018. The time is 04:19:18 and it's week number 04.

(2009-05-25) Kicking ass and chewing spam

... and we're not out of spam yet.

Hormel must feel sad that the name of their product Spam no longer means what it used to for most people. But they point out that their product is Spam with a capital "S" and that unsolicited commercial email (UCE) is spam with a minuscule "s".

I won't discuss Spam here, but rather spam because this is a mail server - not a farm dammit!

Seriously though, how do we remove all of the spam? Well, we don't, but we sure as hell try. There are many ways to combat the flood of unwanted mail, but here's how we do it.

From raw sewage to clean water
"Life is like a sewer what you get out of it depends on what you put into it." - Tom Lehrer

When you build a filtering strategy, try to make the filtering work in stages. You go from coarse to fine-grained filtering. This list is not a comprehensive list nor does it describe the exact order of the procedure. It's meant to show an example of the decision tree involved in filtering spam. This is a list of the methods employed by The ERICADE Network. There are many more controls that you can use, but remember all filters have their merits and flaws!

Stage 0 - Routing
Where: The gateway/firewall
- Do stateful inspection on ingress traffic.
- Filter connections with private addresses in the source addresses coming from the WAN-side.
- Filter connections having your own, internal addresses in the source addresses coming from the WAN-side.
- Filter out Bogon networks.

Stage 1 - Coarse filtering
Where: The MX/SMTP-service
When: On connection
Filters: (Not necessary in this order)
- Tarpit the SMTP response (Wait 25 seconds a before starting to communicate)
- Check that the recipient exists(Well, duh!)
- Check for non-existing FROM-domain.
- Check SPF-record on FROM-domain.
- Check if sending mail server sends to many RCPT TO with non-existing users. This is likely a harvesting attack. We don't allow a SMTP-session to specify more than 5 non-existing users. Or it gets disconnected.

Stage 2 - Black/White-listing
Where: The MX/SMTP-service
When: On connection
Filters: (Not necessary in this order)
- Check if sender (or receiver) is on the system's white list. If either is, bypass all filters.
- Check if sender is on the system's black list. If either is, throw it away.
- Check if sending system is on DNS-black list or not on DNS-white list. (
- Check if URIs in mail is on blacklist (surbl)

Stage 3 - Fine-tuning
Where: Mail server
Filters: (Not necessary in this order)
- Apply local, server wide rules.
- Rank spam-level with bayesian analysis (Spam Assassin). Put ranking on a scale from 1-10 in mail header. Any preexisting rankings must be removed.
- Check if ranking is higher than or equal to the delete threshold (9.5). If it is, delete the mail.
- Check if ranking is higher than or equal to the spam threshold (4). If it is, mark the mail as spam.

Stage 4 - Sorting
Where: Mail server
- Sort spam-tagged mail into the Junkmail folder if user has not disabled this.
- Apply the user's own mail rules.

General considerations
False negatives happen. Some spam manage to get through to your Inbox. It's no big deal. A good remedy is to use the "Mark as spam" feature in your webmail. It tells the bayes-filter to rethink its decision. False positives are cause for greater concern. If the spam-rating is above 9.5 the mail gets deleted and if it's smaller than 9.5 and larger than 4.9 it lands in the Junkmail folder. How often do you check that folder for incorrectly marked mails? Thought so... Therefore The ERICADE Network tries to tune the filters to favor false negatives rather than false positives. A few spam mails getting through is more tolerable than not receiving important email. A good rule of the thumb is that good mail (or ham) should have a negative spam rating and spam should have a high positive spam rating.

Mail delivery as a standard is pretty stable but it is not guaranteed. All SMTP does is to TRY to deliver mail and if it can't it TRIES to inform you that it couldn't.

Other methods
New methods to bypass filters and to improve filters comes along every now and then. It's a true arms race out there, you know. Here's some methods I think are interesting to combat spam.

Grey listing
First time a mail system connects to your mail system it is turned away with a SMTP 4xx code. The 4xx code means that your server has a "temporary failure" and that the sender should return later and try again. This off course is not true. Well behaved systems try later and this time they are let through. Spammers often sacrifice RFC compliance to speed up their "bomb runs" and therefore does not try again later. If they did it would slow them done. In reality they have so many systems that can send Spam, that they actually can afford trying again if Greylisting becomes to much of a problem.
More on Grey listing.

Q: Aren't you giving away information that spammers could use to get spam through to you?
A: Maybe. On the other hand they probably know all the tricks in the book already. Given the very small amount of users this system has, no spammer will gain very much by using this information against us. But the biggest reason not to worry is that no good security has ever been built to rely on "security by obscurity". Remember all the methods we use are pretty much "industry standard". We've not invented any of them.

Posted: 2009-05-10 by Erik Zalitis
Changed: 2009-06-03 by Erik Zalitis

News archive