My mails are being treated like spam! What should I do?

Lately, I've been working on the Butterflive website where users can subscribe. As part of this process, users are sent automated mails (to check if the mail address they provide is correct or not). Alas, my mails were filtered as Spam by my own mail server.

In this article, I'll describe step by step what I did to solve the problem, what was successful and what did not work as expected.

The environment

First, a quick word about the environment:

• Mails are sent in PHP, using the Zend_Mail component of the Zend framework
• The mail agent on the Butterflive server is Postfix
• The mail agent on the machine receiving the mail is Postfix + Amavis and SpamAssassin, on a Ubuntu 9.10 server.

Analyzing the origin of the problem

The first step was to know if my problem was really about getting considered as spam. To check this, I connected on my mailbox server, and checked the logs:

sudo tail -f /var/log/mail.log

Then, I triggered my application to send a mail, and checked what was happening in the logs. I saw this:

Jul 20 10:22:20 dedicated-148 postfix/smtpd[16635]: connect from ks362220.kimsufi.com[91.121.172.133]
Jul 20 10:22:20 dedicated-148 postfix/smtpd[16635]: E5009134057: client=ks362220.kimsufi.com[91.121.172.133]
Jul 20 10:22:20 dedicated-148 postfix/cleanup[16800]: E5009134057: message-id=<20100720074730 [dot] 9E8563AD11 [at] ks362220 [dot] kimsufi [dot] com>
Jul 20 10:22:21 dedicated-148 postfix/smtpd[16635]: disconnect from ks362220.kimsufi.com[91.121.172.133]
Jul 20 10:22:21 dedicated-148 postfix/qmgr[23861]: E5009134057: from=<>, size=68453, nrcpt=1 (queue active)
Jul 20 10:22:24 dedicated-148 amavis[16601]: (16601-14) Blocked SPAM, [91.121.172.133] [91.121.172.133] <> -> <david_negrier [at] thecodingmachine [dot] com>, quarantine:B/spam-BybZjNOxPAu6.gz, Message-ID: <20100720074730 [dot] 9E8563AD11 [at] ks362220 [dot] kimsufi [dot] com>, mail_id: BybZjNOxPAu6, Hits: 7.793, size: 68453, 3448 ms
Jul 20 10:22:24 dedicated-148 postfix/smtp[16804]: E5009134057: to=<david_negrier [at] thecodingmachine [dot] com>, orig_to=<d [dot] negrier [at] thecodingmachine [dot] com>, relay=127.0.0.1[127.0.0.1]:10024, delay=3.6, delays=0.16/0/0/3.5, dsn=2.7.0, status=sent (250 2.7.0 Ok, discarded, id=16601-14 - SPAM)
Jul 20 10:22:24 dedicated-148 postfix/qmgr[23861]: E5009134057: removed

First note: My mail is definitely blocked as spam by Amavis (and therefore by SpamAssassin since SpamAssassin is triggered by Amavis)

Second note: The message-id is somewhat surprising:
message-id=<20100720074730 [dot] 9E8563AD11 [at] ks362220 [dot] kimsufi [dot] com>. I'm sending my mail from a server whose name is supposed to be "butterflive.com". Actually, "ks362220.kimsufi.com" is the name of the machine when it was first installed.

Giving the server the right name

So I decided the first step would be to give my server the right name:

I set the name of the machine to www.butterflive.com:

sudo nano /etc/hostname

Then, I edited the name of the mail server:

sudo nano /etc/mailname

Finally, there is still some work to be done in the postfix configuration:

sudo nano /etc/postfix/main.cf

Edit the my_hostname property:

myhostname = butterflive.com

Then, let's restart postfix:

sudo /etc/init.d/postfix restart

I performed the test again, and here is what I got:

Jul 20 10:25:05 dedicated-148 postfix/smtpd[16796]: connect from ks362220.kimsufi.com[91.121.172.133]
Jul 20 10:25:05 dedicated-148 postfix/smtpd[16796]: D8D8B134057: client=ks362220.kimsufi.com[91.121.172.133]
Jul 20 10:25:05 dedicated-148 postfix/cleanup[16800]: D8D8B134057: message-id=<20100720075015 [dot] 71E253AD11 [at] butterflive [dot] com>
Jul 20 10:25:05 dedicated-148 postfix/smtpd[16796]: disconnect from ks362220.kimsufi.com[91.121.172.133]
Jul 20 10:25:05 dedicated-148 postfix/qmgr[23861]: D8D8B134057: from=<>, size=68438, nrcpt=1 (queue active)
Jul 20 10:25:09 dedicated-148 postfix/pickup[16070]: 2CE0113407E: uid=33 from=<root [at] thecodingmachine [dot] com>
Jul 20 10:25:09 dedicated-148 postfix/cleanup[16800]: 2CE0113407E: message-id=<250ca8da25370b28d33f49b220593358 [at] www [dot] thecodingmachine [dot] com>
Jul 20 10:25:09 dedicated-148 postfix/qmgr[23861]: 2CE0113407E: from=<root [at] thecodingmachine [dot] com>, size=3183, nrcpt=1 (queue active)
Jul 20 10:25:09 dedicated-148 amavis[16601]: (16601-16) Blocked SPAM, [91.121.172.133] [91.121.172.133] <> -> <david_negrier [at] thecodingmachine [dot] com>,quarantine:8/spam-8Dni8JInk6j5.gz, Message-ID:<20100720075015 [dot] 71E253AD11 [at] butterflive [dot] com>, mail_id: 8Dni8JInk6j5, Hits: 7.049, size: 68438, 3692 ms
Jul 20 10:25:09 dedicated-148 postfix/smtp[16804]: D8D8B134057:to=<david_negrier [at] thecodingmachine [dot] com>, orig_to=<d [dot] negrier [at] thecodingmachine [dot] com>,relay=127.0.0.1[127.0.0.1]:10024, delay=3.9, delays=0.21/0/0/3.7, dsn=2.7.0, status=sent (250 2.7.0 Ok, discarded, id=16601-16 - SPAM)
Jul 20 10:25:09 dedicated-148 postfix/qmgr[23861]: D8D8B134057: removed

So I'm still blocked as spam, but at least, the message-id looks better now:
message-id=<20100720075015 [dot] 71E253AD11 [at] butterflive [dot] com>

Looking at other reasons

Since I was still blocked, I headed to the SpamAssassin FAQ

One of the advices was to check that the server time is correct (in order to send mails with a correct timestamp. My server time was indeed wrong by about 30 minutes. I set the server to the correct time, retried... still no luck.

Analysing the mail using Mailing Check program

I kept searching and found a small useful utility program named "Mailing Check": http://www.mailingcheck.com/. This is a program that takes an email in parameter and returns an analysis of the mail (screenshot below).

This is very useful BUT, it requires the mail in ".eml" format. Your mail agent can of course send save the mails you received in .eml format, but that means you must first be able to send the email and not have it blocked by your spam filter. If you can disable your spam filter to perform the test, you should definitely have a look at this solution, since it is really easy to use.

Note: in order to allow SPAM to pass the Amavis filter, use this:

sudo nano /etc/amavis/conf.d/21-ubuntu_defaults

Change the line

$final_spam_destiny       = D_DISCARD;

$final_spam_destiny       = D_PASS;

Then, restart Amavis:

sudo /etc/init.d/amavis restart

After disabling the SPAM filter, receiving the mail and anaylysing the mail... still no luck! Mailing Check tells me the message should not be filtered as spam.

Analyzing the mail headers

However, I've had a nice surprise. When Amavis "$final_spam_destiny" is set to "D_PASS", if a mail is detected as SPAM, it adds a lot of information in the header of the mail.
And here is what I viewed, when looking at the source of the mail (this is the top of the header only):

From - Tue Jul 20 12:08:32 2010
X-Account-Key: account2
X-UIDL: UID156713-1200073226
X-Mozilla-Status: 0001
X-Mozilla-Status2: 10000000
X-Mozilla-Keys:                                                                                 
Return-Path: <>
X-Original-To: david_negrier [at] thecodingmachine [dot] com
Delivered-To: david_negrier [at] thecodingmachine [dot] com
Received: from localhost (localhost [127.0.0.1])
	by thecodingmachine.com (Postfix) with ESMTP id E7391134079
	for <david_negrier [at] thecodingmachine [dot] com>; Tue, 20 Jul 2010 12:11:26 +0200 (CEST)
X-Quarantine-ID: <97zjM71syOfM>
X-Virus-Scanned: Debian amavisd-new at thecodingmachine.com
X-Spam-Flag: YES
X-Spam-Score: 6.676
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.676 tagged_above=2 required=6.31 tests=[AWL=0.372,
	BAYES_00=-2.599, FH_DATE_PAST_20XX=3.188, FH_FROMEML_NOTLD=2.696,
	HTML_IMAGE_ONLY_28=1.561, HTML_MESSAGE=0.001, MIME_HTML_ONLY=1.457]
	autolearn=no
Received: from thecodingmachine.com ([127.0.0.1])
	by localhost (thecodingmachine.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 97zjM71syOfM for <david_negrier [at] thecodingmachine [dot] com>;
	Tue, 20 Jul 2010 12:11:21 +0200 (CEST)
Received: from butterflive.com (ks362220.kimsufi.com [91.121.172.133])
	by thecodingmachine.com (Postfix) with ESMTP id AB891134057
	for <d [dot] negrier [at] thecodingmachine [dot] com>; Tue, 20 Jul 2010 12:11:21 +0200 (CEST)
Received: from localhost (localhost.localdomain [127.0.0.1])
	by butterflive.com (Postfix) with ESMTPS id 9CA9C3AD11
	for <d [dot] negrier [at] thecodingmachine [dot] com>; Tue, 20 Jul 2010 12:07:52 +0200 (CEST)

The interesting line is the X-Spam-Status one:

X-Spam-Status: Yes, score=6.676 tagged_above=2 required=6.31 tests=[AWL=0.372,
	BAYES_00=-2.599, FH_DATE_PAST_20XX=3.188, FH_FROMEML_NOTLD=2.696,
	HTML_IMAGE_ONLY_28=1.561, HTML_MESSAGE=0.001, MIME_HTML_ONLY=1.457]
	autolearn=no

Hooray! This is the ratings Spamassassin gave to my mail, split by spam filters!
So what I find from that line:

• A mail must score below 6.31
• My mail scores 6.676
• AWL=0.372: AWL is Auto white-list. The more I send spam from a domain, the more the AWL will grow. So I'd rather solve my problem quickly before being labeled as a spammer.
• BAYES_00=-2.599: The Bayesian filter has decided that my mail content is definitely not Spam. Thank you Bayesian filter!
• FH_DATE_PAST_20XX=3.188: This one puzzled me a lot. It is supposed to trigger an error if a mail date is in the future. BUT, it is buggy for year 2010 (and at the time of this writing, we are in 2010! Damn!) So I will have to cope with a buggy filter until my mail server is updated.
• FH_FROMEML_NOTLD=2.696: The "From" address is not correctly set in my mail!
• HTML_IMAGE_ONLY_28=1.561: SpamAssassin considers there is a big image in my mail and not enough text.
• MIME_HTML_ONLY=1.457: My mail is in HTML format only and does not have a text only version.

With all these remarks I decided I should start by specifying a From address, and by adding a full text part to may mail (and not only keep that mail HTML only).

And this was enough to finally receive my mail.

Analysing the headers of the mail, I saw this:

X-Spam-Status: No, score=5.881 tagged_above=2 required=6.31
	tests=[BAYES_50=0.001, DNS_FROM_OPENWHOIS=1.13,
	FH_DATE_PAST_20XX=3.188, HTML_IMAGE_ONLY_28=1.561, HTML_MESSAGE=0.001]
	autolearn=no

My mail is no more considered to be a spam, but we have to admin the score (5.881) is pretty close to the target score (6.31).
So improving the mail a bit might be a good thing.

I noticed that new entry:

• DNS_FROM_OPENWHOIS=1.13

Trying to learn more about the rule, I realized the rule does no longer exist in new version of SpamAssassin: http://wiki.apache.org/spamassassin/Rules/DNS_FROM_OPENWHOIS
Therefore, is for the FH_DATE_PAST_20XX, I'll have to ignore that rule.

Conclusion

My mail server is badly misconfigured. Because of FH_DATE_PAST_20XX and DNS_FROM_OPENWHOIS outdated rules, my mails are almost sent to spam. But they are not anymore. And I can be confident that on most host that are correctly configured, I'll get a lower score.

So by adding a text version of my HTML mail, by adding a From clause explicitly in PHP, and by configuring a little better my Postfix settings, I solved my problem, and my mails are not more blocked as Spam. And this is great :)