Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
The Fight Against Spam

By mybostinks in Internet
Fri Aug 31, 2007 at 12:17:28 PM EST
Tags: UCE, SPAM, Yo Mamma Aint Got NoWata, Kuro5hin is dying, K5 is dying (all tags)

Spam: Death By A Million Paper Cuts
The organization I work for receives 19-20 UCEs (Unsolicited Commercial Email) per second, 1.7-2 million potential UCEs per day, 11.7 million UCEs per week. I only have 13,000 email users. These users were desperate and email for many was unusable. Me and another co-worker had 3 months to implement a plan to get rid of most of it. I had to document every step we took and I had one shot to accomplish this. At the end of that time it had to work and it had to be noticeable.

I am not an expert on spam. However, I have learned many things about UCEs and what can be done to fight it and how to adjust to UCEs' dynamic nature.


Our goal was to:
* Replace our aging Linux/Sendmail gateway
* Use a sane and stable MTA (i.e Postfix, Exim, Qmail etc)
* Prevent spammer dictionary attacks
* Block certain countries (country DNSBL) from sending spam and make our domain invisible to new spammers in those countries.
* Accept only email that is RFC 821 compliant
* Use two or three DNSBLs (PSBL, Spamcop and Spamhaus) via datafeeds and local DNS lookups.
* Implement NoListing and Greylisting.
* Minimize false postives and keep them to a manageable level.
* Block as much spam as possible BEFORE any DATA was sent to keep network and server loads sane.
What follows is how we accomplished it. This is not a howto on the subject but I hope it will be useful to anyone that runs a mail server regardless of the size of the organization.

The Problem?: RFC 821
The problem with email/UCEs is the SMTP protocol is very trusting. In fact, it is one of the only protocols that does not require authentication. Later RFCs have enhanced SMTP to have authentication but largely this is not done. Simple Mail Transfer Protocol is simple. Below is all that is required to send an email. First two servers must open a connection. Anyone can do this. Note: S = Sender and R = Receiver.

The Sender opens a TCP connection to port 25 the SMTP port.
telnet mx.example.com 25
R: 220 mx.example.com Simple Mail Transfer Service Ready
S: HELO mx.myexample.com
R: 250 mx.example.com

Once this is completed successfully, the sender begins the rest of the transmission.
S: MAIL FROM:<JoeBlow@example.com>
R: 250 OK
S: RCPT TO:<JudyJones@myexample.com>
R: 250 OK
S: DATA
R: 354 Start mail input; end with <CRLF>.<CRLF>
S: This is where all the email message/body goes
S: ...etc. etc. etc.
S: <CRLF>.<CRLF>
R: 250 OK
Once this happens the Sender is ready to close the transmission.
S: QUIT
R: 221 mx.example.com Service closing transmission channel
Now let's look at an example SMTP transmission on Postfix
Connected to mx.example.com.
Escape character is '^]'.
R: 220 mx1.example.com ESMTP Postfix
S: HELO myexample.com
R: 250 myexample.com
S: MAIL FROM: <root@myexample.com>
R: 250 2.1.0 Ok
S: RCPT TO: <JoeBlow@example.com>
R: 250 2.1.5 Ok
S: DATA
R: 354 End data with <CR><LF>.<CR><LF>
S: Subject: Testing
S: this is a test
S: .
R: 250 2.0.0 Ok: queued as A277539820
S: QUIT
R: 221 2.0.0 Bye
Connection closed by foreign host.
The email is then sent on its way.

RFC 821 was replaced by RFC 1651 which extended the old RFC. Most if not all mail exchangers use the extensions even though the new Standard is backward compatible to the old RFC 821.
R: 220 example.com ESMTP Postfix
S: EHLO myexample.com
R: 250-myexample.com
R: 250-SIZE 10240000
R: 250-ETRN
R: 250-ENHANCEDSTATUSCODES
R: 250-8BITMIME
R: 250 DSN
S: MAIL FROM: <root@myexample.com>
R: 250 2.1.0 Ok
S: RCPT TO: <JoeBlow@example.com>
R: 250 2.1.5 Ok
S: DATA
R: 354 End data with <CR><LF>.<CR><LF>
S: Subject: this is a test
S: This is a test
S: .
R: 250 2.0.0 Ok: queued as 87E9639828
S: QUIT
R: 221 2.0.0 Bye
Connection closed by foreign host.
Why go through all this?
The purpose of showing the above is to show where you want to stop most of the UCEs from entering your network; just before or just after the RCPT TO: command before you receive any data or the bulk of the email. The data stream up to that point would probably never get more than a couple hundred bytes. But once the data stream enters the data command the stream will explode to approximately 3000 bytes without an attachment. If you can stop UCEs before they send their payload you have saved costs in terms of bandwidth and server and network resources. Postfix makes this very easy to do.

The MTA (Mail Transfer Agent)
I have worked for years with sendmail and for the most part hated it. Sendmail is time consuming. The only satisfaction I would get was to get something configured correctly. But sendmail is still a difficult MTA to configure. When Postfix hit version 2.0 I converted to that. Why? Although sendmail is an excellent MTA and for years a "standard" workhorse that moved billions of emails over the Internet, Postfix is much more flexible and configuring it is a more sane task. So for me it was not a difficult decision to go with Postfix. I recommend Postfix or any other MTA over sendmail to anyone. The amount of time you spend learning it is far more rewarding than an equal amount of time learning sendmail. This is probably true with the other non-sendmail MTAs.

In Postfix at the very end of the file but just before smtpd_recipient restrictions add the following two lines. They help prevent spamming by slowing down dictionary attacks and making sure the sender is an 821 compliant mail system. Many spammers and zombie mailers are not. This will knock a few of them out.

smtpd_helo_required = yes - Sender must send a HELO command
disable_vrfy_command = yes - Sender cannot verify that an email address is valid

Postfix comes with built-in anti-UCE mechanisms. I will go over the important ones. There are many and it is not necessary to use all of them. The important ones are the ones placed under the smtpd_recipient_restrictions
. When you place options here order is important. If you mess up here you can cause your system to be an open relay. While using these you can test each one by using the warn_if_reject before each command, like so: warn_if_reject,
reject_unauth_destination,
Then you are now able to look in your maillog files and see reject_warnings to see the effects it would have had they been in effect.

Here are the basic anti-UCE controls Postfix uses:
reject_invalid_hostname - many non-legit senders issue nonsensical hostnames in the helo or ehlo stage so get rid of them. On the other hand, RFC 821 compliant mailers announce exactly who they are so we'll let them through.
reject_non_fqdn_hostname - This also checks RFC compliance. It should look like mx.example.com.
reject_non_fqdn_sender - Reject the request when the MAIL FROM address is not in fully-qualified domain form, as required by the RFC.
reject_non_fqdn_recipient - Reject the request when the RCPT TO address is not in fully-qualified domain form, as required by the RFC.
reject_unknown_sender_domain - Reject the request when Postfix is not final destination for the sender address, and the MAIL FROM address has no DNS A or MX record, or when it has a malformed MX record such as a record with a zero-length MX hostname. All legitimate mail exchangers should have MX records. Many spammers and zombies do not.
reject_unknown_recipient_domain - Similar to the one above, reject the request when Postfix is not final destination for the recipient address, and the RCPT TO address has no DNS A or MX record, or when it has a malformed MX record such as a record with a zero-length MX hostname.
permit_mynetworks - Now it is OK if it is from anywhere in my network.
reject_unauth_destination - * Postfix is mail forwarder: the resolved RCPT TO address matches $relay_domains or a subdomain thereof, and contains no sender-specified routing (user@elsewhere@domain), * Postfix is the final destination: the resolved RCPT TO address matches $mydestination, $inet_interfaces, $proxy_interfaces, $virtual_alias_domains, or $virtual_mailbox_domains, and contains no sender-specified routing (user@elsewhere@domain).

If you noticed and have referred to the legitimate email transmission above, you will see that all the above controls will reject email before it sends its payload when it hits the DATA command. On days when I get blasted by spammers the above directives kill up to 10% of the spam alone. Obviously, this is not enough but it is low cost and necessary.

The next restrictions you want to use are block lists. This can be a very slippery area and not one to take lightly.

Why Block Lists?
I once hated them. They produced many false positives and for the most part were operated by questionable people. Getting off a list was nearly impossible for some and seemed irrational. I had at one time been a victim of spamcop.net. In short, I hated block lists. Some time ago Al Iverson started dnsbl.com and is an excellent resource reviewing and analyzing the various block lists. His criteria is simple; percentage of accuracy and percentage of false positives. The higher the accuracy and the lower the false positives the better the overall rating of the list.

An RBL (Realtime Block List) is like having a staff that does nothing but checks reports of spam. Very large commercial ISPs do in fact hire a number of people that do just that. Google, Yahoo, MSN and AOL have people on duty that check spam reports. I don't have that luxury and probably 99% of you don't either.

There is a downside to using a block list. There are lots of them. You have to consider what will happen if you use them. Even though Spamhaus and lately Spamcop have extremely low false positives you may still have a problem or two occaisionally. Don't overly rely on them. They will block lots of spam but unless you use extremely aggressive lists spam will get through. No one method works but a combination of methods to block spam will get rid of most of it. Don't rely entirely on them.

I chose only 3 block lists...none are real aggressive: Spamhaus Zen (includes sbl, xbl and pbl), Spamcop and PSBL(Passive Spam Block List). I chose these because of their extremely low false postives and their high percentage of accuracy, according to dnsbl.com. Because of the volume of the email that we receive, I had to subscribe to the above lists in particular Spamhaus. If your email volume is not more than 1000s per day then it should not be necessary. In any case it is a bargain.

In Postfix in the section smtpd_recipient_restrictions and after the permit statement, we place the RBLs like so:
reject_rbl_client bl.spamcop.net,
reject_rbl_client zen.dnsbl,

The first statement above will look up the IP address of the sender at bl.spamcop.net. If it is on their list then it gets blocked after the MAIL FROM command. The second statement above however will not work until you install and configure BIND 9.x and rbldnsd. More about this in a minute.

Blocking Countries: YMMV
This is real controversial for a lot of email administrators and many administrators do not do it. I do not recommend it unless you do a lot of analysis of your log files and do a comprehensive check of your business rules. If you do not take the time to do this analysis then don't block countries at all or you will be sorry. This is especially true if e-commerce is a critical part of your business. YOU HAVE BEEN WARNED.

I spent about a month analyzing log files to determine where most of our spam was coming from and I developed a "Top 10" list of countries and checked them against the Whois registry.

Next I wanted to minimize my bandwidth to the Internet so I installed BIND 9.x and rbldnsd to do local lookups of my RBLs and Blocked countries list that I had compiled. I wanted all the RBLs and blocked countries to be looked up locally and I accomplished this by creating new zones. The idea is to use named to do all lookups and for the dnsbl zones (spamhaus, spamcop and countries I wanted to block) named then forwarded the lookup to the rbldnsd name server. This required that I install two name servers. However, rbldnsd is very low on server resources, fast and efficient. It is perfect for these kinds of look ups.

At this point, everything is complete for Postfix. You can put at the very end of the restrictions the following:
permit - Everything else moves on to the next stage.

So far, Postfix and block lists have done a majority of the work. But we still have a couple more methods to consider and use.

Nolisting and Greylisting
I decided that I could try nolisting or what some call "poor man's greylisting". Nolisting and Greylisting relies on your public DNS and its MX records. In my DNS I had the following in my BIND data file:
example.com. IN MX 10 mx1.example.com.
example.com. IN MX 20 mx2.example.com.
What the above means is that a sending mail server will first try to send mail to the the mx1.example.com. server first. If it can not do this it will try to send mail to mx2.example.com. Nolisting means that one or the other mail servers is never available. In fact, the server that is the nolisting server doesn't accept mail delivery and it doesn't need to exist. With nolisting you have to conduct some tests to see which server should be the nolisting server will be the most effective. There is some debate whether spammers skip the primary mail server and send right away to the secondary mail server. The thinking is that the secondary mail server will be a less protected server and spam will have easier entrance into your network. For this reason, I have greylisting on the secondary server.

Greylisting is a method of stopping spam by refusing the sender the first time it tries to send email to any user and the receiving server requesting that the sender send it again at a later time. Here is how it works, it is called a 'triplet'.

1. The IP address of the host attempting the delivery
2. The envelope sender address
3. The envelope recipient address

If receiving MTA has never seen this triplet before, then it refuses this delivery and any others that may come within a certain period of time with a temporary failure. This works very well and generally is not noticed by the administrator or user. It is VERY effective against attacks from zombies.

The basic greylisting software for Postfix is Postgrey. Postgrey uses the DBM database and I used it for several months. It is easy to set up and use and comes with a nice report feature called Postgreyreport. There are several packages that you can use for Greylisting. I chose SQLGrey because it has more options to configure and to look at. It uses either MySQL, Posgresql or SQLite. This allows for a lot of flexibility.

I decided to make my primary server a nolisting server and have the secondary mail server (running Postfix) use Greylisting. As a general rule zombie mailers and a lot of spammers will try one and never come back to try again. According to the RFC if a mail server is down the sending mail server should try again some time in the future. Many spammers won't do this because it is not efficient for them to do so. Greylisting takes advantage of the fact that spammers want to spew as much spam to as many users as possible. Retrying to send email is not efficient for them and greylisting takes advantage of this.

What Comes Next?: Spamassassin and ClamAV
At this point you have stopped 70-80 percent of UCEs coming into your network. Even though this is a great improvement it is not good enough...not even close. Our internal groupware servers consisted of a mail hub and 3 "post offices" that the hub routes users' email to. We had an excellent 3rd party commercial application that had heuristics for spam and also did virus scanning.

What you do next depends on your resources. ClamAV. Spamassassin and the alternative Spambouncer will be needed for two reasons: to get rid of embedded URIs that carry dangerous payloads, virus laden email attachments, for heuristics and Bayesian filtering.

You have a choice here you can send the mail on to an internal mail hub or if your server is beefy enough you can process mail that gets through with Spamassassin. Postfix handles this very well. The only problem is Spamassassin is a resource hog on your server. Keep this in mind. For awhile I put Spamassassin and ClamAV on a mail hub inside our network and processed the mail before it was sent on its way. Later I had a gateway server outside our network that I ran it on an it only gets stressed when there are spam blasts.

With heuristics we are able to get rid of 90-95 percent of the UCEs that enter our network. This makes UCEs manageable. I still strive for 100% UCE free.

Other anti-UCE measures
Powerful anti-UCE tools of interest that you might consider using.
SPF-Sender Policy Framework
: The main aim of SPF is to prevent forged email. This is done using DNS TXT resource records. It determines if a mail server is authorized to send email to your domain or not. For a small to medium sized business where you have lots of control over your users AND you have lots of UCEs this is an excellent option. For us it is still out of the question and would require formal training for each user. In my situation this is not possible.
DomainKeys
DomainKeys does not prevent abuse but makes it easier track. That fact alone kept me from considering it. It verifies the source and content. It is a form of authentication.
HashCash
I didn't consider this one either. Although like the one above if most MTAs used it I would as well. With the volume of mail we receive it would require considerable computational resources I didn't want to expend. May be useful to a smaller number of users.

I am sure there are others I have forgotten or didn't seriously consider.

Image and currently pdf spam has not been a great problem but one that I need to address. I do that on the mail hub.

Log files and Reports
If you do all the above and then forget about your work, chances are good that eventually spam will start to build up again. Log files and reports are the tools that will help make adjustments as spam changes. Without them I would be lost. Looking through Gigabytes of files that my maillog generates would be mind numbing. I have installed pflogsumm which generates a very large and detailed file and also wrote a script that gives me exactly what I want. Remember you will always be a step behind.

A Typical But Light Day
249158 Turkey
206287 Poland
126605 SpamHaus xbl
63588 SpamHaus pbl
40026 Germany
32108 Russian Federation
25596 GREYLISTED
22245 Korea
20887 RFC - Need fully qualified hostname
17376 France
16844 Brazil
16693 China
15808 Message accepted
11987 Taiwan
11833 Spain
7957 Argentina
7954 Israel
7564 Italy
7108 Czech Republic
7060 SpamCop bl
6599 Netherlands
6109 Hungary
5390 Romania
5309 RFC - Domain not found
5106 Japan
4744 Surriel bl
3230 Chile
3109 Bulgaria
3015 Vietnam
2403 Belgium
1776 Ukraine
1587 Slovenia
1548 United Arab Emirates
1495 RFC - Helo Invalid Name
1433 SpamHaus sbl
1432 Greece
1040 South Africa
618 Relay access denied
608 Senegal
567 Estonia
544 Ivory Coast
435 RFC - Need fully-qualified address
392 Saudi Arabia
293 Malta
106 RFC - Malformed DNS Server
70 RFC - Improper pipelining
36 Marketers
13 Nigeria
8 Benin
6 Kenya
5 SURBL
1 Greenland
1 Botswana

Henny-penny...
A couple of days after we cut spam down to almost nothing, I was walking down the hall, returning to my office to troll on Kuro5hin and met a co-worker from another department.

"Is the mail server down?" she inquired. She had a worried look on her face. "I haven't received much email today."

I smiled to myself when I realized that she was no longer spending half the morning deleting spam from her inbox.

"No", I replied, "The sky's not falling."

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Kuro5hin
o Google
o dnsbl.com
o Henny-penn y...
o Also by mybostinks


Display: Sort:
The Fight Against Spam | 48 comments (41 topical, 7 editorial, 0 hidden)
i don't know what your problem is (2.73 / 19) (#1)
by circletimessquare on Thu Aug 30, 2007 at 12:33:55 AM EST

my penis is now 30 inches long, i refinanced my mortgage at 2% interest, i got $30 million from my friend in nigeria after forwarding him $30,000, these lesbian teenage orgies are HOT, and i'm marrying my gorgeous russian bride tomorrow


The tigers of wrath are wiser than the horses of instruction.

Custom tailored pants? -nt (3.00 / 2) (#5)
by Kasreyn on Thu Aug 30, 2007 at 04:49:29 AM EST

nt
"Extenuating circumstance to be mentioned on Judgement Day:
We never asked to be born in the first place."

R.I.P. Kurt. You will be missed.
[ Parent ]
ROR! (3.00 / 3) (#10)
by mybostinks on Thu Aug 30, 2007 at 07:58:06 AM EST

poignant and hilarious CTS!

[ Parent ]
This brings back memories (2.75 / 4) (#2)
by gndn on Thu Aug 30, 2007 at 12:34:56 AM EST

I remember trying to admin a Slackware box, years ago, that was running a mail server and getting quite a bit of spam (though nowhere near as much as what you're discussing here). Postfix made it fairly easy to clean it up to a manageable level, even without using SpamAssassin. In fact, I noticed a huge drop in spam just be putting some sanity checks on the HELO command - most spammers would issue a HELO with my own domain name or IP address, which obviously is invalid. I also screwed around with something I think called smtp_verify_sender or some such, I can't remember exactly, with fairly good results. It basically issues a "try again later" response when mail comes in from an unrecognized sender, then attempts to verify that sender's existence by connecting to their mail host and querying them. Didn't work all the time, but stopped quite a few at the gate.

I'm quite happy I no longer have to deal with all that crap. I switched that domain to use gmail for email instead of hosting it myself and noticed a colossal drop in spam - whatever Google is doing, they're doing it very very well.

+1 from me, even though I just skimmed most of it.

I agree (none / 0) (#42)
by tetsuwan on Wed Sep 05, 2007 at 08:10:06 AM EST

My google account gets at most one (1) spam email per week nowadays. And I haven't heard of any false positives (= people trying to reach me but failing to do so).

Njal's Saga: Just like Romeo & Juliet without the romance
[ Parent ]

this story might be useful (2.20 / 5) (#6)
by th0m on Thu Aug 30, 2007 at 05:23:24 AM EST

but it is also incredibly fucking boring

This is K5 (none / 0) (#15)
by Cro Magnon on Thu Aug 30, 2007 at 01:51:36 PM EST

It's supposed to be boring.
Information wants to be beer.
[ Parent ]
Never block a reply! (3.00 / 4) (#7)
by Elija on Thu Aug 30, 2007 at 06:21:36 AM EST

Do you also process outgoing e-mail?

If you do, then you should probably whitelist addresses that your users send e-mail to. You should probably also recognise replies coming from a different address (using the quoted Message-ID).

It is extremely annoying when someone sends you an e-mail asking for some info, you send a reply and then a get a message saying your reply has been blocked because it looks like spam. A correctly formed reply to a private message should always be recognised as such and should never be blocked.

Another thing to bear in mind: sometimes people forward a spam message, either as part of an abuse report or because it was amusing. Make sure your spam filters won't stupidly block such messages by naively looking at the words in the body. At the very least, don't do what many companies have done and have an "abuse@..." address that refuses messages that contain forwarded spam.


Outbound email... (none / 0) (#11)
by mybostinks on Thu Aug 30, 2007 at 08:35:14 AM EST

Yes we do but outbound mail is strictly controlled on our switches and routers via ACLs. In version 2 that I am working on, I will be implementing what you describe.

[ Parent ]
Much easier solution: (2.75 / 8) (#12)
by uid 71137 on Thu Aug 30, 2007 at 08:47:48 AM EST

I see all your incoming mail claims to be from the same domain. Just drop all email from example.com. It's not even in use, as it's a reserved domain. See RFC 2606.

DAMN, I KNEW IT WAS SOMETHING (3.00 / 2) (#13)
by mybostinks on Thu Aug 30, 2007 at 08:53:05 AM EST

AS SIMPLE AS THAT!

[ Parent ]
Another tactic (3.00 / 3) (#14)
by i1n3k on Thu Aug 30, 2007 at 12:00:20 PM EST

One technique I found to be surprisingly effective in cutting down spam for a site I used to administer was SMTP transaction delays.  20 seconds is enough for most zombies to give up, and legit senders don't care.  With even an initial delay, zombies will start pipelining before such capability is advertised, which is actually a protocol violation, and many MTAs will then drop the connection due to sync error.

You handle a lot of mail, though, so it might not be such a great idea of this specific case.  More here: http://www.tldp.org/HOWTO/Spam-Filtering-for-MX/smtpdelays.html

Libenter homines id quod volunt credunt. — Julius Caesar

I think I've heard of this as some (none / 0) (#16)
by thugsonfilm on Thu Aug 30, 2007 at 05:17:33 PM EST

sort of "tar pit" like defense.

[ Parent ]
sounds like tarpitting (none / 0) (#17)
by mybostinks on Thu Aug 30, 2007 at 07:57:08 PM EST

which I have/am considering.

[ Parent ]
i moved to vote /nt (none / 0) (#18)
by mybostinks on Thu Aug 30, 2007 at 09:28:06 PM EST



HashCash (none / 1) (#22)
by Vs on Fri Aug 31, 2007 at 03:55:03 AM EST

Note that *checking* HashCash doesn't cost much and can be done inline by eg. SpamAssassin. The point of HashCash is to be (computationally) expensive for the sender.

I think it's not widely used by senders anyway.
--
Where are the immoderate submissions?

I used to get about 600 spam messages a day (none / 1) (#23)
by xC0000005 on Fri Aug 31, 2007 at 09:15:02 AM EST

until the latest filter went into place. Now between the server junk filter and the client one I rarely see it. I agree about the country blocks. If you don't have partners that can vastly cut down on SPAM all by itself. Selling them to management can be a pain "But what if someone in afghanistan wants to mail me?" but they DO work. RBLs - well, depends on the RBL. The big problem with greylisting is that a lot of spam isn't a direct transfer. Yeah, the "other guys" should stop it as well but that doesn't mean they will, and you wind up penalizing legitimate MTAs at times.

Voice of the Hive - Beekeeping and Bees for those who don't
There is no perfect solution (none / 1) (#24)
by mybostinks on Fri Aug 31, 2007 at 10:25:05 AM EST

and at best if you can kill off 90% of the spam you are doing great.

Each solution I use has its drawbacks and collateral damage I am under no illusion about that. With the number of users I have and the mess email was in when I arrived, collateral damage was entirely acceptable.

I think until authentication is implemented everywhere on every server it will continue to be a problem.

[ Parent ]

I know. (none / 1) (#25)
by xC0000005 on Fri Aug 31, 2007 at 10:55:17 AM EST

I work in an evironment that moves volumnes of mail larger than what you list and it's a never ending battle. As you've said, the key is stopping the SPAM before the DATA command. Preferably before the IP connect. Body based filtering is fine as a second tier solution but I don't want to ever accept the message to begin with. I was the victim of a reverse NDR attack a couple years ago and finally wound up writing rules to nuke NDRs to me. I think I counted around 20,000. Some idiot in east asia was at the root of it.

Voice of the Hive - Beekeeping and Bees for those who don't
[ Parent ]
Those MTAs are broken (none / 0) (#46)
by FattMattP on Sat Sep 15, 2007 at 12:27:56 PM EST

The big problem with greylisting is that a lot of spam isn't a direct transfer. Yeah, the "other guys" should stop it as well but that doesn't mean they will, and you wind up penalizing legitimate MTAs at times.
Greylisting doesn't do anything that a normal MTA wouldn't do. An MTA might legitimately be too busy to handle requests and send a 4xx response to the client telling it to return later. All greylisting does is send that message to clients it has never seen before. If an MTA isn't honoring the 4xx response and trying again later then it's not RFC compliant and is broken.

[ Parent ]
This is true (none / 0) (#47)
by mybostinks on Sun Sep 16, 2007 at 04:00:03 PM EST

Nolisting and Greylist have proven to be very effective with my set up. The Nolisting server receives 2.5-3.0 million hits per day and is my primary MX record. The greylist server which is the secondary MX record only receives 1 million hits per day. At least a half million never return.

[ Parent ]
sweet article (none / 1) (#26)
by loteck on Fri Aug 31, 2007 at 12:32:01 PM EST

breath of fresh air.
--
"You're in tune to the musical sound of loteck hi-fi, the musical sound that moves right round. Keep on moving ya'll." -Mylakovich
"WHAT AN ETERNAL MOBIUS STRIP OF FELLATIATIC BANALITY THIS IS." -Harry B Otch

Further down the Chain (none / 1) (#27)
by frankwork on Fri Aug 31, 2007 at 01:40:17 PM EST

I've had incredibly good luck with a program called DSPAM. It's somewhat resource intensive (particularly with disk space), but then I haven't done all that much tuning with it. It's also a fairly far-downstream approach, in that the mail is already on your server by the time DSPAM shuffles it into a junk mail folder. And training it (depending on your setup) is a bit more of a chore than some of the client-side filters. That said, it works like fucking magic. Once it's trained, the error rate (both false positives and false negatives) is miniscule.

just greylist with spamd and pf from openbsd (none / 1) (#28)
by Nark on Fri Aug 31, 2007 at 03:45:03 PM EST

as per:

http://www.openbsd.org/spamd/
and http://www.greylisting.org/implementations/openbsd_spamd.shtml

I originally wanted to use (none / 0) (#29)
by mybostinks on Fri Aug 31, 2007 at 03:57:54 PM EST

this. I again am considering this with version 2.

[ Parent ]
we cut spam way down using spamd and pf (none / 1) (#31)
by Nark on Sat Sep 01, 2007 at 07:52:26 AM EST

my advice is to just forward using pf and log for 2 - 3 weeks. Then make up your whitelists from the pf logs. spamd works really well if you have good ( and well kept ) whitelists.

Watching spammers waste time trying to deliver mail is very satisfying.


[ Parent ]

SPF and SA (none / 1) (#30)
by ebonkyre on Fri Aug 31, 2007 at 06:00:52 PM EST

SA doesn't seem to have a rule for "no SPF record" (maybe newer one does, ours doesn't) so I wrote this one:

ifplugin Mail::SpamAssassin::Plugin::SPF
meta SPF_NOT_FOUND      ((SPF_PASS + SPF_FAIL + SPF_SOFTFAIL + SPF_HELO_PASS + SPF_HELO_FAIL + SPF_HELO_SOFTFAIL + AWL + ALL_TRUSTED) < 1)
describe SPF_NOT_FOUND  From untrusted source with no SPF record
score SPF_NOT_FOUND     1.0
endif


The truth hurts sometimes... Nothing beats a nice fat cock. ShiftyStoner
Couple more (none / 0) (#32)
by pw201 on Sat Sep 01, 2007 at 01:55:59 PM EST

What I do, as I user who's pulling mail from my host using fetchmail, not running my own SMTP server: scan with Spamhaus Zen and the Distributed Checksum Clearinghouse (which catches a lot of stuff, but you must whitelist legitimate bulk email from mailing lists, so your users may not tolerate it).

If you're running your own server, I'd recommend blocking hosts with no rDNS (just 4xx temporary failures in case it's an intermittent DNS problem) or whose rDNS contains the couple of least significant bytes of their IP address (eg cab62-1-89-88-150-161.dsl.club-internet.fr for the IP 89.88.150.161, from my latest received spam). The latter is a sign of generic IP space from a consumer ISP: you don't want email from them, they're probably zombies, even if they've not turned up in the Spamhaus XBL or PBL yet.

These days the plan has to be to differentiate from the good side of the net from the bad, whether that badness is malevolence or incompetence. That's what the Spamhaus PBL (now by far the most effective DNSBL I use) and the DNS checks are getting you (as well as the greet_pause feature in recent sendmail: if Postfix has an equivalent, use that). People who want to send legit email get themselves a real, RFC-compliant, server with with a non-generic rDNS. A business might have to communicate with some clients who are in the clueless zone, but that's what a whitelist is for.

Slightly simpler idea... (none / 1) (#35)
by The Vast Right Wing Conspiracy on Sat Sep 01, 2007 at 09:43:01 PM EST

Catch and greylist email that either:

  1. Has four dots in the hostname
  2. Has four dashes in the hostname
  3. Doesn't resolve


___
I'm a pompous windbag, I take myself far too seriously, and I single-handedly messed up K5 by causing the fiction section to be created. --localroger

[ Parent ]
Re: (none / 1) (#45)
by mikael_j on Thu Sep 06, 2007 at 06:17:03 PM EST

A bit of a problem with blocking all mail from servers with "generic" rDNS names is that, as an example, my ISP doesn't allow you to change the rDNS even for the basic business service, you have to pay for the monitored service where they place a Cisco router on-site and hand you a block of IPs otherwise every IP you get even if it's static IPs will have a rDNS along the lines of 4-3-2-1.wholesale.tier2isp.tld which means my mail server would be blocked unless I was willing to fork over another $100/month or so for the next service tier just for rDNS and a few silly things like ten free windows antivirus software licenses...

/Mikael
We give a bad name to the internet in general. - Rusty
[ Parent ]

Either your math is wrong or I misunderstand (none / 0) (#33)
by The Vast Right Wing Conspiracy on Sat Sep 01, 2007 at 08:20:36 PM EST

19-20 emails per second is not the same as 2 million/week. You're off by a factor of 10.

___
I'm a pompous windbag, I take myself far too seriously, and I single-handedly messed up K5 by causing the fiction section to be created. --localroger

I will have that fixed (none / 0) (#37)
by mybostinks on Sun Sep 02, 2007 at 08:11:45 AM EST

if i can get the admins' attention.

Thanks

[ Parent ]

More thoughts: (3.00 / 2) (#34)
by The Vast Right Wing Conspiracy on Sat Sep 01, 2007 at 08:45:57 PM EST

Some of this is just in general, not specifically addressed to you or your article, since someone will eventually find this on google...

Do NOT bounce spam email if you're running an after-queue filter (which you almost certainly are if your userbase is larger than you and your dog.) I'm sick of bounces from mailservers run by asshats who couldn't deliver spam that spoofed my domains. If you're running a before-queue filter you can reject as part of the SMTP session. That's cool.

Do NOT bounce virus email that contains the virus in the email. This should be common sense, but apparently it's not.

Use anvil. Anvil is your friend. You can use it to do fun things like limit the number of emails a client can deliver per unit of time and more. This is great for cutting down on spamming mxes (tend to burst a hundred or more emails at a time) as well as finding abusive users on your end.

Be careful with RFC-strictness. There are a lot of broken mailservers out there, and some of them are important. For example, RIM's mailservers (fucking blackberries...) used to treat temporary failures from greylisting as permanent and bounce all email from blackberry users. Not cool.

p0f is a useful tool. It plugs into amavisd-new, you can use it to score Windows clients higher, for example.

___
I'm a pompous windbag, I take myself far too seriously, and I single-handedly messed up K5 by causing the fiction section to be created. --localroger

spoofed domains (none / 1) (#36)
by Zombie Gautama Buddha on Sun Sep 02, 2007 at 12:56:22 AM EST

If there is one thing more annoying than spam, it's having your domain used as the sender in spam and getting their bounced e-mail.

[ Parent ]
thoughts (none / 0) (#38)
by Cruel Elevator on Sun Sep 02, 2007 at 01:46:36 PM EST

Greylisting is a godsend.

Putting this on the postfix main.cf seems a good idea:

reject_unknown_reverse_client_hostname

However, this appears to block legitimate mail from people who forgot to setup rDNS. I couldn't use it at the end.

I have used SASL auth for users not on the local network with good results. Recommended.

SPF would have been useful if more people implemented it. So far very few do. Took it out at the end with no measurable difference in traffic.

I'm not a fan of postfix after-queue spam filtering. Once the mail is in the system, it's a lost game. You sacrifice your server's CPU cycle. You can pass the job to the client - then you trade off with the server's diskpace and bandwidth. At the client end, the client uses Thunderbird's anti-spam features or POPFile.

if you have a high volume mail server, postfix's before-queue filtering is impractical. So there's no clear answer on how to go on about it.

Razor, PyZor or DCC seemed like a good idea to start with. But they seem to perform poorly compared to spamassassin, and also have a high false positive rate. Not good... this could've been a solid before-queue filter.

I've given up virus scanning on the mail server - wastes too much of resources. Blocking lame VBS, PIF and EXE solves a part of the problem.

Controlling UCE is extremely challenging. There are a lot of trade-offs, and there are too many wankers out there who would not make their mail servers standards compliant. That's not a problem - the problem is that your users expect legitimate mails from those servers. Sigh.

C.E.

I don't like client-side filtering (none / 0) (#39)
by Delirium on Sun Sep 02, 2007 at 06:41:47 PM EST

You can't rely on filtering via Thunderbird and such if you want to give your users a webmail option so they can check their mail from public kiosks and so on. And for people like me who use multiple clients (I check my mail from a home desktop, work desktop, and laptop), maintaining separate client-side filters on each annoying.

[ Parent ]
when (none / 0) (#40)
by /dev/trash on Tue Sep 04, 2007 at 06:14:56 PM EST

my ISP implemented Greylisting.  The spam in my inbox tripled.

---
Updated 02/20/2004
New Site
sounds like they put you on the (none / 0) (#41)
by jangledjitters on Tue Sep 04, 2007 at 06:37:00 PM EST

spam lovers list.

hi
[ Parent ]
I gave up and let Gmail do the work (3.00 / 2) (#43)
by GReaper on Wed Sep 05, 2007 at 09:56:49 PM EST

I've tried my best to tweak my own server to avoid as much spam as possible, but eventually I decided to take the easy route and use Gmail thanks to Google Apps.

Postfix tweaking, SPF, greylisting, clamav, spamassassin, amavisd-new, the list was far too much.  The setup wasn't too bad, however the latest deluge of PDF spam came along and I just gave up.  Yes I probably could've tweaked the settings to add extra rules for PDF spam, but chances are something else would come along after which requires yet another change in the setup.

So I now have the pleasure of allowing Google to be my front end mail server.  I'll carry on using POP3 as always so I don't become too reliant on a web interface, although the backup is always useful if I'm not on my home machine.  If something goes wrong with Google then I can always go back to hosting my own mail, but for now this seems like the easiest thing.

They are excellent... (3.00 / 2) (#44)
by mybostinks on Wed Sep 05, 2007 at 11:38:09 PM EST

but the cost for us is prohibitive with their non-free service. Even with what they pay me I do more than just admin email. Most of our business centers around our groupware. They considered outsourcing email and it was determined by the powers that be that it was not a workable solution.

[ Parent ]
blocking countries (3.00 / 2) (#48)
by Tjalfi on Mon Sep 17, 2007 at 12:09:50 AM EST

I've found it quite effective to block large sections of Afrinic, Lacnic, and Apnic. We don't have customers from outside the state, let alone internationally. Here is a partial list from memory: 59/7, 60/7, 41/8, 196/8, 189/7, 200/6. I haven't spent much time on spam filtering recently because it has ceased to be a problem.
"With energy and sleepless vigilance go forward and give us victories." - Abraham Lincoln to Major General Joseph Hooker, 1863
The Fight Against Spam | 48 comments (41 topical, 7 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!