Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Fighting spam at the server level

By Cluster in Internet
Wed Dec 28, 2005 at 11:49:54 PM EST
Tags: Internet (all tags)
Internet

The last few evenings and nights were spent learning about the architecture of Postfix mail transfer agent, the workhorse responsible for routing and delivering email on a high percentage of the Internet's mail hubs, including mine.

The focus of my research has been controlling e-mail spam. My goal is to stop over 95% of spam that audaciously attempts to lodge itself in my inbox. As a result of my hijinx, I have not received a single spam message in the last four days, although there have been many attempts that were rejected by my all-new, all-souped-up spam filtering system.

I will describe my efforts, along with pertinent background information, for posterity.


The beautiful theory of early 1980s

First, let's discuss how spam thrives. The de-facto email protocol, SMTP, has been around since early 1980s, and was designed when Internet users were very trusting of others, and the very idea of spam probably hasn't occurred to anyone.

Here is the lifetime of a typical email message before the rise of anti-spam solutions, and how the designers of SMTP envisioned it:

  1. Using a mail client such as Mozilla Thunderbird, Kristy writes me a message and clicks 'Send'.

  2. Her mail client connects to her ISP's (say, Comcast's) mail server, tells it from whom the message is and where it's going, and gives it the message itself.

  3. Comcast's server accepts the message and leaves a mark on it, stating that this message was received from Kristy's computer by Comcast's systems at X o'clock.

  4. Comcast's server looks at where this message needs to go, and makes a connection to that host. For example, if the destination is someuser@qnan.org, it connects to mail.qnan.org because that's where I configured mail destined for qnan.org to go.

  5. My server accepts the message from Comcast and leaves another mark on it, stating that this message was received from Comcast by qnan.org at Y o'clock. My server then delivers the message to my mailbox.

  6. I use a mail client to access my mailbox and read the message. I can also see the complete path of this message, from Kristy all the way to me, which theoretically allows me to complain to Comcast if Kristy is spamming me.

Like a cheap hooker, this setup is easy-peasy and bursting with opportunities to exploit it. It has also been a virile vector for viruses and undesired hints that you don't measure up and require penile enhancements. Let's see why.

Exploitations; a.k.a. The End of Innocence

Today, the biggest problem is "zombie" computers that have been hijacked by trojans, viruses, or other badness to do various nefarious tasks without the owners' knowledge. A very popular nefarious task is--surprise!--spamming. On a regular DSL connection, a regular PC can attempt to deliver up to 10,000 messages per minute.[source] The zombie machine goes down its list of addresses, tries to connect to the mailserver associated with the next address, and if connects, it delivers the message. If it doesn't connect, it just goes down the list.

Another big problem has been open relays. These are well-intentioned mail servers that do not check whether a user is authorized to use the server--they just blindly accept , which allow Pavel-- another Comcast user--to dump his email message in step 2 on AOL's mail server instead of on Comcast's. This might not be a big deal for normal users, but spammers, for whom 10,000 messages per minute are a norm, may cause a serious starvation of resources (Denial of Service) for mail servers that were not configured for such aggressive behavior.

Worse, let's say that AOL sees this behavior and blocks Pavel after he has sent 100,000 messages to random Internet users. No problem; he simply and instantly switches to spamming through Southwestern Bell's mailservers.

China has been historically bad when it comes to open relays: it is a safe haven for spammers due to an enormous number of misconfigured servers that are set up by amateurs and never fixed. Many system administrators in the U.S. have even claimed support for entirely blocking mail from China until the Chinese fix their problems.

One reason that open relays are worse than zombie computers is that they follow mail standards: they insist on trying to deliver mail, including spam, for up to 5 days per message.

And of course, Pavel wouldn't want to be caught spamming from i_am_pavel@comcast.net or whatever his real email address is--that's too traceable. Instead, he'll cleverly use "From: Bob Guccione <bob@penthouse.com>" since that will likely draw more buyers for penile enhancements. And guess what? Until recently, mail servers were happy to oblige with this obviously fake header.

Or like email viruses of the last few years, he could pick random people that know each other, and email Person X supposedly from Person Y to instantly gain attention and trust. The mail protocol is happy to comply.

So, what has the Internet intelligensia come up with to combat this decade-long problem? I will list only the solutions that I've implemented on my server, but this list sufficiently covers the breadth of all of them.

Modern solutions

First, the number of open relays (not zombies, but well-intentioned but misconfigured mail servers) is reducing due to greater attention to security both from software programmers and administrators. Now, if Chris is traveling, staying in a hotel room, and wants to send a message through his work's mail server, it is no longer sufficient simply to throw the message at any random mail server--all well-behaving mail servers now require that users that are not explicitly within their network (such as for ISPs) authenticate prior to having their messages accepted. The most common method of authentication is SASL, which requires a username and password pair for every authorized user. The less-common method, but one which I use, is TLS, which issues eah authorized user an identity certificate that allows Chris to prove his identity to the mail server.

Now, onto the actual message. The following solutions are ordered by the order in which they are triggered as a new message arrives.

At the forefront of protection is the oldest and still a very effective solution is the use of Realtime Blackhole Lists, which are actively-maintained lists of IP addresses that are known to send spam. When a message arrives, the receiving mail server queries the list: "Do you have any dirt on the sending IP address?" If the list was made aware of recent malicious activity, the mail server will reject the message. If Maegan's machine acquires a virus and becomes a zombie for a spammer, it's entirely possible that just one hour later many mail servers will begin rejecting messages from her computer as well as from all other computers that share her IP address. This block may be in effect for as long as 6 months! For more information about Blackhole Lists, visit Spamhaus which "tracks the Internet's Spammers, Spam Gangs and Spam Services, provides dependable realtime anti-spam protection for Internet networks".

Next, the mail server may check what's called a Sender Policy Framework, which allows mail server administrators to specify exactly which IP addresses are allowed to send mail from a certain domain name. In the example above, Pavel pretended that his spam came from penthouse.com. If the administrator of that domain name specified that only the IP address 38.117.136.143 can claim to be from that domain name, Pavel's message would be rejected. This can be very effective, but is not yet configured by many domain names. Eventually (and theoretically), Pavel will ONLY be able to specify a Comcast email account as his "From" address, as all other domain names will refuse to honor his IP address.

A system of a similar ilk which was developed by Yahoo within the last few years is called DomainKeys. This relies on the tried-and-true concepts of public/private key cryptography, and has strong similarities to PGP. The idea is that every incoming message is signed by the domain name from whom the message claims to be, using a private key. The receiving system then retrieves the public key of that domain name and checks whether it can verify the message. If it cannot, or if the signature is missing entirely, then the receiver cannot be sure that the message was sent from the domain name from which it claims to be sent, since only the authorized mailserver of that domain has the private key needed to sign the message.

Sender Policy Framework and DomainKeys are systems that focus less on the question of whether a message is spam or not and more on whether the sender can be held accountable for the message, no matter what it is. Imagine--if you had to leave a piece of paper with your home address each time you vandalized a building with graffiti, it might greatly cut down on your desire to vandalize.

If the message passes all the checks above, we can be sure that our incoming message is not being sent by a known zombie machine, and it is addressed truthfully. Now time to see whether it is a zombie machine which is yet unknown, or a well-behaved mail server.

One of the most recent concepts which is very rapidly becoming popular across the Internet is called greylisting. This solution takes advantage of the fact that the SMTP standard was designed with the ability to defer a message in case the receiving mail server is down, overloaded, or temporarily misconfigured. This causes the sending mail server to keep it on its system just a little while longer while waiting for the receiving server to heal itself. Again, a well-configured and well-intentioned server may keep a message for up to 5 days while waiting for the next hop to become available. Greylisting takes advantage of this by playing hard-to-get for unverified senders, mimicking a temporary disability, and watching whether the sending server will retry or give up. If the server gives up, it's safe to assume that it was a zombie machine. Most well-behaving servers try again within 15 minutes.

Now we know that the incoming message is not sent by a zombie, and it is addressed truthfully. We can now accept it for delivery. But, is it well-transmitted garbage?

The last, most computationally-expensive, and arguably the most effective, hoop through which a prospective 'ham' must jump through before being left alone in the user's inbox, is the merciless content inspection by SpamAssassin. It is a set of tests that check the message for all sorts of tell-tale signs of being spam. For example, do you mention "rolex"? Do you use tricks such as "v1 a g r4"? Does your email address end in numbers? Do you use poorly-formed HTML? Do you use large fonts or poor contrast? Each of these increases the probability of your message being spam. Once a certain threshold is reached, your message is kicked down the drain.

To decrease the chance of your spammy-looking message from being deleted by SpamAssassin, a sender can use a system called HashCash, which works like postage stamps: it adds a "cost" to sending a message, except this cost is measured in energy and time. It requires that the sender spend at least several seconds calculating something the correctness of which only takes milliseconds to verify. This has the capability to reduce a spammer's output from 10,000+ messages per minute to maybe only 30 that have a chance of passing SpamAssassin.

Now realize: some or all of these advanced systems are working across the Internet for every message you send and receive.

What can I do to aid the battle?

If you control a mail server, the best thing you can do for yourself and your users is to implement some or all of these systems. The technology is there. It is strikingly effective. It has very few downsides. The ball is in your court.

If you own a domain name, then regardless of whether you run mail or not, configure a Sender Policy Framework! This will prevent spammers from getting away with spoofing their mails with your domain name.

If you are a user... are you still getting spam? Is it more than just a few messages per day? Contact your mail provider and demand (or ask politely, if your mail is free) that they implement some or all of these systems.

Spammers are slowly adapting to new weapons, but I am certain that the hive of bright minds against spam will continue outpacing them.

(Note: this is a slightly-modified version of my recent LiveJournal post.)

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Spam is best solved
o by perfecting mail filtering 13%
o economically, by requiring HashCash or equivalent 13%
o by implementing and enforcing anti-spam laws 30%
o by educating society on how to treat spam 26%
o by apathy 16%

Votes: 30
Results | Other Polls

Related Links
o Yahoo
o Postfix
o e-mail spam
o Mozilla Thunderbird
o source
o open relays
o identity certificate
o Spamhaus
o Sender Policy Framework
o DomainKeys
o greylistin g
o SpamAssass in
o HashCash
o my recent LiveJournal post
o Also by Cluster


Display: Sort:
Fighting spam at the server level | 62 comments (52 topical, 10 editorial, 0 hidden)
HashCash (none / 0) (#2)
by t1ber on Tue Dec 27, 2005 at 10:20:28 AM EST

I personally don't like HashCash.  On the surface, it looks like a Good Idea, in all reality the way around it as I see it is find a botnet, make each member of the botnet generate a stamp, then have them send the stamps back to you and you send the mails.  While Joe Spammer might have all the bandwidth in the world for sending Viagra spam, it's true the clients do not.  Making the clients generate the stamps is trivial once they're zombied, and exchanging the stamps is costing merely bytes of bandwidth.  

James Spammer, who is an even bigger fish then Joe Spammer, is just going to buy a bunch of barebones systems and expand his crunching.  Since PCs are around $100 or so for a barebones system, if you slow him down by a factor of 10, he's just going to buy 10 more machines.  The arms race just keeps on going until such a time when entry-level spamming is too expensive for anyone to professionally do.  The result will be that spamming becomes the profession of plenty of people who still live with their parents and play D&D in their basement.  The high level spammers will move to nations that don't have laws against spam and the living is cheap.   Then what, blacklist those guys?

Instead of HashCash, I humbly suggest Mail Frontier ( http://www.mailfrontier.com ).  They had a rocky startup -- I found a few security problems with the client when it first came out -- but their guys are quick to help with support and re-engineering the product should anything really serious come along.  I was impressed with them when the company bought their product.  It works like a captcha:  Strange e-mail addresses are sent back a URL to examine.  The URL displays an image and asks the user to answer a question about the image.  Questions are similar to "How many kittens are in this image?" and the image will be of three kittens and a dog.  Mails that have no response, bounce, or answer incorrectly are ignored.  Mails that answer correctly are forwarded along.  Someone is going to say, "OH NOS, IT DOES NOT WORK IN TEH LINUX!!!1!eleven".  If you're that concerned about security and having a *nix mail gateway, you should probably consider *BSD.

(This started out as editorial until it got to a few paragraphs, d'oh!  I'm not trying to start flamewars in the edit queue).

And she said...
Durka Durka Mohammed Jihad
Sherpa Sherpa Bak Allah
Hadji girl I can't understand what you're saying.

HashCash, C/R systems, etc. (none / 0) (#4)
by Lisa Dawn on Tue Dec 27, 2005 at 01:48:40 PM EST

I've always been suspicious of the voluntary sender-cost systems, mostly because none of those I've investigated are very strong against pre-generated hashes, and tracking spent hashes poses problems. I'd rather see a system that accepts your message, and provides a computational challenge (on the spot), lets the connection close, and will approve the message once it receives another connection with the correct result. From the same host. Within a given limit. Etc. (Diagnostic messages can always ride on message IDs.)

Your description of MailFrontier sounds like a standard human-interactive challenge/response system. I think these systems are fine for personal addresses, sites, whatever, but I'd never install one in a server-wide configuration. Some sending systems are incompatible with C/R systems. Additionally, you can compound a DDoS with one of these.

[ Parent ]

DDos (none / 0) (#7)
by t1ber on Tue Dec 27, 2005 at 03:43:46 PM EST

Some sending systems are incompatible with C/R systems. Additionally, you can compound a DDoS with one of these.

If you're doing something incompatible such as listserv or similar, you can whitelist the address.  As for DDoS, there is a rate limit on the responce e-mails so that -- to humans -- the mail is challenged in a reasonable legnth of time.  To computers, 10 seconds or so is forever.  Dig around the site a bit if you're looking for some enterprise material, they do have PDFs you can read.  I'll field any questions from my own use of the product if you're interested.

And she said...
Durka Durka Mohammed Jihad
Sherpa Sherpa Bak Allah
Hadji girl I can't understand what you're saying.

[ Parent ]

Understood (none / 0) (#20)
by Lisa Dawn on Wed Dec 28, 2005 at 10:44:05 AM EST

My objection to the C/R messaages is that they involve another connection. If you're subject to a DDoS and your resources are pushed to the limit, sending additional messages out, be it now or 10 seconds from now, is still going to push up your load, and probably your queue as well.

I can see configurations that would lessen the impact, and maybe I shouldn't be all that worried about the worst case scenario: things are going to be a nightmare, regardless. From there, my attention shifts to customers. I'd gladly turn on a C/R system for a specific domain to quiet someone who thinks the spam they're receiving is somehow my responsibility. (Of course, I make it my responsibility to add value to the service.)

I may make a recommendation on C/R systems in the future, as a value-added service, if I can make it cost-effective. That future is after implementing more fundamental facilities in the mail server.

[ Parent ]

NO! (none / 0) (#44)
by DavidTC on Fri Dec 30, 2005 at 10:27:52 AM EST

C/R systems are spam. The only ethical way of implimenting a C/R system is via the SMTP reply, and no currently available system does that, probably because quite a few systems do not actually display the reply to the sender and thus the system would not work very well.

All existing systems generate a new message and send it to the forged Mail From the spammer is using.

So all a C/R system does it redirect mail to innocent people who already are getting bogged down under outdated systems that accept and bounce instead of reject. We've spent years of effort reducing such accept and bounce in favor of rejection, and C/R people think they can come along and do exactly the same thing for personal gain. As long as they don't get the message, sending it to someone else is fine.

It personally makes me a little sick.

And me, and a quite a few other people serious about fighting the spam problem in ethical ways, have made it a personal vow that if you redirect spam to us, we will confirm the message so that you, too, can see the nice fancy spam you directed to us. And we won't confirm it if we did send it, because we frankly don't want to talk to such amoral people.

If you want to make people jump through hoops to email you, fine, it's your email account. That doesn't give you the right to send random messages to what are almost always forged addresses. That would make you a spammer.

-David T. C.
Yes, my email address is real.
[ Parent ]

HashCash (none / 0) (#30)
by nossy on Thu Dec 29, 2005 at 12:51:06 AM EST

Even simpler, can't Joe Spammer precalculate hashcash hashes set in the future, and *then* spam a mailserver with 10k mails?
"The difficult thing with programming isn't usually the syntax but the fact that most people can't cope with the computer doing exactly what they tell it to do."
[ Parent ]
Yeah... (none / 0) (#31)
by Cluster on Thu Dec 29, 2005 at 01:09:55 AM EST

Sure, but that would still require the same amount of computation, and thus prep time.

[ Parent ]
exactly (none / 0) (#41)
by user 956 on Fri Dec 30, 2005 at 12:05:50 AM EST

James Spammer, who is an even bigger fish then Joe Spammer, is just going to buy a bunch of barebones systems and expand his crunching. Since PCs are around $100 or so for a barebones system, if you slow him down by a factor of 10, he's just going to buy 10 more machines. The arms race just keeps on going until such a time when entry-level spamming is too expensive for anyone to professionally do. Exactly. HashCash and other systems are slightly-useful cruft addons for a system that was never designed with any type of this abuse in mind.

The problem with coming up with any new system, of course, is that you're going to have a bunch of companies all chomping at the bit to push some proprietary BS as part of the replacement system.

What needs to happen is the development of a new open standard, much like the development of ipv6.

(though regardless, the pessimist in me believes that any technology or system developed will always have exploits)
---

Top Chuck Norris Facts.

(lazy sunday)
[ Parent ]
Internet Mail 2000? (none / 0) (#60)
by grendelkhan on Sun Jan 08, 2006 at 08:18:44 AM EST

An interesting idea, though horribly named, would be Dan Bernstein's Internet Mail 2000 design. Jonathan de Boyne Pollard has fleshed it out greatly here. While "IM2000" is possibly the worst name anyone could have chosen for this project--which, I'm relatively certain, was started after the year 2000--it may be what you're looking for in terms of a system designed by someone not interested in, as you put it, "proprietary BS".
-- Laws do not persuade just because they threaten --Seneca
[ Parent ]
Oh Great (none / 0) (#51)
by KaBewM on Fri Dec 30, 2005 at 10:42:35 PM EST

It works like a captcha: Strange e-mail addresses are sent back a URL to examine. The URL displays an image and asks the user to answer a question about the image. Getting users to stop clicking random shit they recieve in e-mail will help the fight against spam. If this recieves widespread acceptance, expect spam to grow to higher levels of acceptance since users will just clicky click the links they've been told not to.

[ Parent ]
excellent (none / 0) (#8)
by circletimessquare on Tue Dec 27, 2005 at 04:34:13 PM EST

thanks author, you're a benefit to us all

you're kind of like the cdc telling us all how not to get syphilis.. seems kind of boring and useless, unless you consider that without the cdc, we'd all have syphilis

i do control an smtp server and nntp server on the open internet, and i am no expert

and even if i were, this kind of stuff is always useful, no matter how knowledgeable you consider yourself to be, because the pace of technological change is such that no one can claim they are an authority on subjects like this for very long without a refresh, a refresh exactly like this article


The tigers of wrath are wiser than the horses of instruction.

regardless of content, prose is atrocious. -1 % (none / 1) (#9)
by creativedissonance on Tue Dec 27, 2005 at 04:59:10 PM EST




ay yo i run linux and word on the street
is that this is where i need to be to get my butt stuffed like a turkey - br14n
Thanks (none / 0) (#10)
by blackpaw on Tue Dec 27, 2005 at 06:49:20 PM EST

I admin a small server on the side for our company. This was perfect for my level of expertise - I will setup domain keys for our domain now.

WIPO (3.00 / 2) (#14)
by destroy all monsters on Tue Dec 27, 2005 at 08:11:24 PM EST

Public execution of spammers.

"My opinion: You're gay, a troll, a gay troll, or in serious need of antidepressants." - horny smurf to Lemon Juice
WIPO (none / 0) (#16)
by Smothie on Tue Dec 27, 2005 at 10:25:10 PM EST

IAWTP

I was going to answer, "With high velocity lead poisoning" but your answer is close enough.

--

Please visit my scoop site, Guppylog - For help with all livebearing fish.
[ Parent ]

yeah (none / 0) (#57)
by user 956 on Sun Jan 01, 2006 at 10:50:26 AM EST

I was going to answer, "With high velocity lead poisoning" but your answer is close enough.

Preferably the hollow-tipped variety. In the gut.
---

Top Chuck Norris Facts.

(lazy sunday)
[ Parent ]
Adequate. (none / 1) (#17)
by jd on Wed Dec 28, 2005 at 12:12:16 AM EST

There are some omissions. Spamming started with a bunch of lawyers who were drumming up business by mass-posting to USENET and then who wrote up all the techniques they knew on mass-mailing over the Internet as a book. Despite a lot of effort by concerned Internet users, the book was actively promoted by a lot of the same people who - today - are bemoaning the problems spamming is having.

The next problem is connecting to the mail transfer agent. This is usually a raw socket connection over port 25. The protocol for SMTP (and ESMTP - the version used today by most mail servers) is very simple and can be manually typed in, or cut-and-paste.

This has one major benefit for spammers. Because you are not sending via your own mail server, there is nobody to disconnect you. Because the FROM address you give is (usually, if you do this) fake, there is nobody to complain to. You can't trivially block such messages, because (E)SMTP does not restrict the path a message takes to get from A to B. Blocking breaks the protocol which is generally not a good thing.

Domain Keys are ok, but (E)SMTP-over-SSL has the same effect with the added benefit that the mail is safe from packet sniffers. It would be better yet if users were encouraged to digitally sign their e-mails, as signing by the domain doesn't prove the user is who they say they are, only that the domain is.

It would also be good if sites that suffered phishing a lot (eg: eBay, PayPal, Amazon.com, etc) would not only digitally sign their e-mails at the user level (or at least the domain level), but if they also actively encouraged ISPs to simply drop all unsigned e-mail that claims to be from their respective sites.

Anonymity is an important part of the Internet, but there can be special "anonymous" digital signatures. That's not a problem. The problem is in forged identities, not anonymous ones, and it is a problem that could very easily be solved.

So how does one go about solving it? For the end user, I would strongly advise using an e-mail client that supports either X.509 certificates and/or PGP/GPG signing. X.509 is more common, but getting a meaningful certificate can be a lot harder and are expensive if they're to be worth it. PGP/GPG keys can be countersigned in a web of trust and cost nothing.

For the MTA admin, I would suggest prioritising mail over IPSec connections, then over SSL connections, and only then sending raw mail. If the site is big, you would definitely be advised to get a commercial-grade X.509 certificate. Not sure if there are any MTAs that support PGP/GPG keys.

I would also strongly suggest that admins mix all types of categorization for e-mails. Don't just use one method - mix-n-match for the best results. Remember that the most specific rules should always come first and that rules specific to an interface should only be applied to that interface.

(eg: If your internal network uses a private numbering scheme, like 10.x.y.z, then obviously no computer inside of that network can have a "global" address. Therefore, if your server gets a mail message from a global address from your internal network, no matter how real it looks it must be fake.)

Oh, and if you use a virus scanner in the MTA, don't have it look at the file's extension to determine what type it is. There are tables your computer can use which look for telltale indicators in the file, which are going to be much more accurate. Use those.

Finally, if you are in a corporate network, have TWO MTAs - one in the DMZ, one on the inside. Mail on the inside should not be blocked by a heavy load on the outside server (whether inbound or outbound). I've seen too many places use just one massively-shared mail server for everything and it invariably ends up a pool of molten aluminium and silicon.

Okay. (none / 0) (#46)
by DavidTC on Fri Dec 30, 2005 at 10:53:02 AM EST

You mean well, but a lot of that doesn't make much sense.

Signing emails won't do anything to reduce spam. If that becomes a popular method to whitelist on, spammers will just start signing email. Or just fake signing...mail servers don't have the time to track down the right PGP key to see if the signature is real or just random text appended to the message. Same with X.509 as far as I know. The X.509 key they got might be signed by someone important, but the mail server does not have the key to check this. Although admittedly I don't know exactly how X.509 works, but I am assuming it is like SSL.

And it is completely impossible for a message to come from an external IP on an internal interface in almost every network. The mail server would respond to the router, who would route the signal out onto the net, and those the spammer wouldn't get any further than the connect. (Not that I know why a spammer would be on internal network.)

While egress and ingress filtering is important, it really doesn't do anything for actual TCP/IP connections. It stops DoSs via single TCP packets and UDP. But no one can fake a real TCP/IP connection if they aren't getting the other end of it, and certainly can't send mail over it. (It would make blacklisting rather moot if they could.)

Spammers do use the lack of filtering to send spam, but they do it by getting dialups, and buying a T3 with no ingress filtering, so they can send spam out with the T3, using the address of the dialup. Any return packets go there, but because it's the same machine they can continue the converstation over their T3. Basically the exact opposite of how one-way satellite used to work. They get the outgoing bandwidth of a T3, and if anyone tracks and blacklists their IP or gets it cancelled, well, that was a cheap dialup.

However, no one can do anything about that, as the lack of filtering is on the T3. Only the phone company can fix that.

And good mail virus scanners should look at all files, period. Just because that file is a JPG doesn't mean they should let it have an executable virus in it, even if no one can figure out how it's going to replicate.

However, you certainly should have two MTAs if you have an internal network. Or, at least, one MTA that actually does the mail and one that does the blocking and hands the mail off to the internal system if it passes various checks, with internal users talking purely to the internal one and sending mail from there.

-David T. C.
Yes, my email address is real.
[ Parent ]

Gah. (none / 0) (#47)
by DavidTC on Fri Dec 30, 2005 at 11:02:40 AM EST

They buy a T3 with no egress filtering.

No one ever buys anything with ingress filtering, or even worry about it, as that doesn't really matter. Anyone who wanted to send a packet to you with an address of 10.0.0.1 from outside the network would need the complicity of your service provider or reprogram their router, so worrying about your service provider having 'ingress filtering' at the router is a bit dumb.

Ingress filtering is only ever used at the peer points of different ISPs, as far as I know.

-David T. C.
Yes, my email address is real.
[ Parent ]

Check your facts.. (none / 0) (#61)
by deep44 on Sat Jan 21, 2006 at 10:35:39 PM EST

Domain Keys are ok, but (E)SMTP-over-SSL has the same effect with the added benefit that the mail is safe from packet sniffers.
DomainKeys and ESMTP TLS do not have the "same effect"; in fact, they're barely even similar. If you're not sure what I mean, I would recommend starting with a full review of the OSI model.

[ Parent ]
Goodness... (none / 1) (#19)
by starX on Wed Dec 28, 2005 at 10:25:33 AM EST

I don't run a mail server, but I often times wind up providing tech support to the people who do.  It never ceases to amaze me the number of people who have open spam relays and start wondering why their unit is running slow, they've been blocked by aol, etc.  This is good stuff, I'm voting it up so that I can send some of these guys here.

"I like you starX, you disagree without sounding like a fanatic from a rock-solid point of view. Highfive." --WonderJoust
Not too good article, imho. (none / 0) (#21)
by arcade on Wed Dec 28, 2005 at 03:17:36 PM EST

Technical articles tend to either be very good - or quite shoddy. This one came in the quite shoddy category in my book. The reason is that there is far to many errors.

A few of your errors:

  1. The 80s. Most boxes were unix boxes and many did their own relaying..
  2. SASL is not username/password authentication. SASL is what you describe as TLS. TLS is just an encryption protocol, where you can send .. guess what .. username and password!
  3. Your claims about SPF is inaccurate. If a spammer buy a domain (which they often do) they can point their spf records at comcast mailservers - and spf will not stop it. spf does forward-lookups on domains, not reverse lookups on ips.
I'm afraid I've given you -1, due to inaccuracies.

--
arcade
Thanks (none / 0) (#22)
by Cluster on Wed Dec 28, 2005 at 04:12:30 PM EST

Thanks for the feedback.  Let's see if we can come to an agreement.

1.  I am not sure I understand what your argument is.  Yes, many did their own relaying, but that does not invalidate any points.  There is still a matter of an end-user depositing the mail onto whichever server allows them to do so, regardless of whether they do it remotely or using a local Unix account.

2.  It's true that TLS is an encryption protocol, but Postfix allows it to be used for authentication, by having the server administrator generate and issue client certificates which end-users present to the server.  This guarantees to Postfix that the user is authorized and is allowed to relay, while at the same time establishing a secure tunnel.  SASL does not require client-to-server encryption -- I can authenticate via SASL and then transfer my message in plain text.

3.  Yes, the fact that a spammer owns a domain name gives him greater freedom to spam, but the accountability is still there.  If the spammer <i>has</i> to use his own domain name to spam (since other domain names will refuse to honor his IP address), it is that much easier to identify him and shut him down.

[ Parent ]

Hmm, we're both right (and wrong) about number 2. (none / 0) (#32)
by arcade on Thu Dec 29, 2005 at 02:53:14 AM EST

1. I am not sure I understand what your argument is.

My argument is quite simply that the original thought was not that you had to relay through your ISPs email server, you could run your own. It was one of the flaws in the original idea. :-)

Now onto where I did a faux pax in my criticism, by not fact-checking my own "knowledge" first:

2. It's true that TLS is an encryption protocol, but Postfix allows it to be used for authentication, by having the server administrator generate and issue client certificates which end-users present to the server. This guarantees to Postfix that the user is authorized and is allowed to relay, while at the same time establishing a secure tunnel. SASL does not require client-to-server encryption -- I can authenticate via SASL and then transfer my message in plain text.

TLS is quite simply "Transport layer security". It doesn't require a client side sertificate in itself. It may be enabled - and I've always thought SASL was the name of that - which it turns out it isn't. SASL is a mechanism for agreeing on authentication and security, if I've read myself up to scratch. It seems that you may indeed use plaintext authentication with SASL, or you may use kerberos .. or you may use other mechanisms.

3. Yes, the fact that a spammer owns a domain name gives him greater freedom to spam, but the accountability is still there. If the spammer has to use his own domain name to spam (since other domain names will refuse to honor his IP address), it is that much easier to identify him and shut him down.

The problem is that he doesn't have to use his own domain name. He may still take over large numbers of machines and just use some email address that is accepted by the local mail-server of the victim .. or he may use his own domain, which quite a few spammers seems to be doing these days.

SPF doesn't help to reduce spam. It helps to reduce joe-jobs - which of course is important, but it's not really a spam prevention tool.



--
arcade
[ Parent ]
Please enter a subject for your comment. (none / 0) (#34)
by FieryTaco on Thu Dec 29, 2005 at 12:48:09 PM EST

Saying "TLS is quite simply 'Transport layer security'." is like saying "the NSA simply listens to stuff." True, but so over-simplified as to be completely incorrect.

TLS gives you encryption of your communications. It gives you the ability to know who you are talking to. It provides the ability to prove that you are who you claim you are. You can validate that not only are they who they claim they are but they have also proven so to a supposedly independant third party. You can choose not to talk to a partiuclar person any more. It is quite a bit more than the basic definition of the acronym.

[ Parent ]

Was this in the edit queue? (2.66 / 3) (#23)
by sudog on Wed Dec 28, 2005 at 05:51:59 PM EST

I didn't notice it there.

Your article has some errors. Well.. lots of errors. First and foremost, spam only works because it works. I.e. a profitable fraction of people who receive spam actually answer it and pay money to the people sending it to them. If we were to wipe this fraction of the internet away in a big cleansing, spam would disappear shortly thereafter.

Back in the 80s, the Internet as we knew it then was actually deliberately non-commercial. Also, the vast majority of people were aware enough to know that spam was a bad thing, and that encouraging it by paying spammers money was folly. Therefore, back in the early days of the Internet it was most likely commercially infeasible to spam.

Also, in all your examples, you should be using @example.com instead of real domains. Now those servers are going to be hammered with requests. In a way, you're doing more harm than good, because most people who are interested in this already know everything you wrote about, and because you've just redirected spammer attention at Comcast. Heck, I don't see a single original observation or idea in that whole piece..!

So why didn't you write an original article for us? People are going to vote you down because this isn't original--it's a repost. And .. from .. ewww Livejournal of all places. Yuck man!

Anyway all of those techniques you list are doomed to failure. SPF sucks. DomainKeys suck. And stupid stuff like this: "If you control a mail server, the best thing you can do for yourself and your users is to implement some or all of these systems. The technology is there. It is strikingly effective. It has very few downsides. The ball is in your court." Well, that's just plain inflammatory. I run none of those systems, and quite frankly I never will. And you know what? Everyone who wants to send me email can do it from any machine (misconfigured or anonymous) that they want to.

The only viable solution to 100% block all spam--period--is to use a system of one-off aliases, punish harvesters with abuse complaints, and.. well.. essentially implement what I wrote in my own anti-spam article, here:

Spam Free At Last.

In other words, the users who want to be spam-free have to take a little responsibility for their own mailboxes and stop frittering around with silly half-functional anti-spam programs.

For the record, I'm still 100% spam-free and have been since I implemented my Final Solution.


Different strokes (none / 0) (#25)
by Cluster on Wed Dec 28, 2005 at 06:08:19 PM EST

The main difference between my methods and yours is that mine are designed to be implemented entirely server-side, not requiring end-users to "take responsibility".  They are end-users, and they simply cannot be expected to be as responsible with their email addresses as your scheme requires them to be.  Furthermore, there is no reason for this when server-side systems are so effective.

SPF and DomainKeys don't "suck", they just have not been implemented enough to make a significant impact.  If it is so difficult to get knowledgeable and experienced system administrators to implement these schemes, how can you possibly expect end-users to take the necessary initiative to implement your solution?  I cannot.

Of course, doing both would be the best of both worlds; then any spam that makes it past the server can still be dealt with through thorough controls of who has what email alias.  That requires far more knowledge, care, and willpower than 99% of email users are willing to offer.

Regarding the LiveJournalness of this article... what is the problem?  I had initially written it for my Linux Users Group, and my LiveJournal is syndicated on our Planet Planet.  If I had written the article for K5 first, and posted it on LJ later, would that be more acceptable?  This instinctual dislike of anything LJ doesn't make sense.

[ Parent ]

Yo. (none / 1) (#26)
by sudog on Wed Dec 28, 2005 at 07:16:15 PM EST

SPF and DomainKeys do suck. :-)

SPF has a very limited scope and is essentially irrelevant in preventing spam. In fact, it would be trivial to provide zombie botnets with the ability to send spam from zombie-helpful subdomains, and then the problem ends up being one of blacklisting anyway.

Additonally, DomainKeys sucks because it doesn't do anything about spam. Its stated purpose is to prevent the exploitation of trust: NOT spam itself. That's irrelevant and is primarily aimed at reducing actual fraud. Real spam isn't fraudulent in nature. People will still be exploited and turned into zombies, and spam will still be delivered.

As I stated in the preamble, my anti-spam technique is primarily for people like us: the end-users can fend for themselves. Quite frankly, it's pointless to try to help them because we simply can't impart our knowledge and expertise effectively enough to protect them without in-depth and constant contact or instruction. Therefore, I don't expect end-users to take the necessary steps to implement my solution.

It's not an instinctual dislike of Livejournal. It's an instinctual dislike of a repost from what we consider to be a lesser site. However, a complete repost is bad form to begin with: buy an ad, or make your story a mindless link. Otherwise, you're just being redundant.. right?


[ Parent ]

SPF and potential blacklisting (none / 0) (#28)
by AngelKnight on Thu Dec 29, 2005 at 12:14:50 AM EST

True.  However, proper deployment of SPF (which is, in itself, possibly a pipe-dream, which I don't argue) can help cut down the number of domains you're forced to blacklist from (potentially) every domain on Earth to just about 10,000 jokers who suck.

[ Parent ]
True enough. :-) (none / 0) (#36)
by sudog on Thu Dec 29, 2005 at 05:59:44 PM EST


I just don't buy the idea that it will cut down on spam itself.

[ Parent ]
SPF wasn't designed to stop spam (none / 0) (#58)
by FattMattP on Tue Jan 03, 2006 at 07:05:17 PM EST

SPF has a very limited scope and is essentially irrelevant in preventing spam.
SPF was never designed to stop spam. It was designed to stop joe-jobs.

[ Parent ]
I know. (none / 0) (#59)
by sudog on Thu Jan 05, 2006 at 12:43:25 PM EST

The person I was replying to implied it did.


[ Parent ]
this is an mlp (none / 1) (#27)
by the77x42 on Wed Dec 28, 2005 at 08:35:22 PM EST

A better-written article would include why spam really works, i.e., because there is actually a market for it.

Section at best.


"We're not here to educate. We're here to point and laugh." - creature
"You have some pretty stupid ideas." - indubitable ‮

My 2c (none / 0) (#29)
by AngelKnight on Thu Dec 29, 2005 at 12:19:28 AM EST

I have reasonable luck with greylisting.  At least this way, my spam tends to utterly restrict zombie PCs, from which the vast majority of my spam seems to originate.

I get maybe 4 or 5 AOL/hotmail/yahoo spam e-mails each day, which is annoys me considerably less than than at least 30 per day.

The nice thing about greylisting is that I'm usually not that inconvenienced.

The potential main disadvantage of greylisting is jackass clients who believe that e-mail should be a now-now-now thing instead of what it is: store for purpose of inspection.

One day, personal computers will operate at a speed of around ~1,000,000 MIPS and have 128GB of accessible core, which makes greylisting less useful.  All-else-equal, zombie PCs might really go through the trouble to store up temporarily-deferred spam at that point :(

Solution (none / 0) (#48)
by DavidTC on Fri Dec 30, 2005 at 11:17:35 AM EST

Three levels of greylisting for each address.

One level is none at all.

One level is all the time.

One level is making greylisting only enabled for business hours. Well, technically, you want to disable it at about eight in the morning, and enable it about seven or so. And don't forget timezones. This is great, because probably 75% of spam is sent outside business hours, because spammers have a better chance of not getting a complaint, or having any sort of manual intervention done about them, then.

This can do the first two for postfix, using the policy daemon framework. It stores in a mysql database, so it's trivial to write a PHP frontend to it.

And I'm no coding expert, but I was able hack the time thing in fairly easily.

And that thing lets you blacklist sites that try to send to a user defined list of invalid email address. I had thought of something like that, but always worried about typos. But here, you can go and specify addresses that have never existed, or only existed for a short time in a wildcard address, that spammers try to send to. As you add them manually, you can just look to make sure they aren't a typo of a real address people might make.

Some people are going 'Huh? That mail can't get in anyway.'. But if you know what I'm talking about, you know. I get over 50% of total mail addressed to complete gibberish addresses. Now if they try to send to those addresses, that IP gets blocked for a few days, which keeps them from sending to the valid ones they have. And every time they try for another invalid one, the time limit gets reset.

Use the spammer's dirty lists against them.

-David T. C.
Yes, my email address is real.
[ Parent ]

I swear. (none / 0) (#49)
by DavidTC on Fri Dec 30, 2005 at 11:44:37 AM EST

I have to stop posting for a while, that's twice I've said exactly the opposite of what I meant.

I mean, enable greylisting only for non-business hours, obviously.

-David T. C.
Yes, my email address is real.
[ Parent ]

uh, here's a little news about what an intro is (1.60 / 5) (#33)
by tert on Thu Dec 29, 2005 at 10:48:03 AM EST

You use it to write "I think I'm pretty smart, and covering lots of new material, but in fact I'm just using RBLs, greylists, and spamassassin."

Then we know from the get go that you have nothing useful to say, and we don't read your article.

Thanks for nothing infantile author.

This isn't slashdot.  We don't get all gooey just because someone is posting a noob-targetted intro to what we do all day.

Go post this shit at slashdot (1.00 / 7) (#35)
by bighappyface on Thu Dec 29, 2005 at 02:18:18 PM EST



IAWTP - they like shitty prose there % (none / 1) (#38)
by creativedissonance on Thu Dec 29, 2005 at 06:56:54 PM EST




ay yo i run linux and word on the street
is that this is where i need to be to get my butt stuffed like a turkey - br14n
[ Parent ]
The hunger for gay cock can never be satiated (1.00 / 13) (#37)
by Ken Pompadour on Thu Dec 29, 2005 at 06:37:44 PM EST



...The target is countrymen, friends and family... they have to die too. - candid trhurler
Well... (none / 0) (#39)
by Cluster on Thu Dec 29, 2005 at 10:59:47 PM EST

If it's not aggressive enough to get past greyfiltering, we don't want it.

[ Parent ]
question (none / 0) (#40)
by user 956 on Fri Dec 30, 2005 at 12:00:12 AM EST

This makes me wonder. There's an interesting article over at the Wall St Journal about the National archives' job to archive the ~100 million emails sent and received during the last 2 presidents' terms (Clinton & Bush).

I wonder what percentage of that is SPAM, and if the white house uses any kind of filtering?



---

Top Chuck Norris Facts.

(lazy sunday)
i tried (3.00 / 3) (#42)
by frozencrow on Fri Dec 30, 2005 at 02:47:10 AM EST

i tried really hard to like this article. i did. i even made myself stop and notice things like that you (correctly) identified zombies as the biggest source of spam these days. but i just couldn't ignore the stupid shit, like the fact that you then went on to talk about open fucking relays. open relays! yes, they're still out there, but no, they're not a big problem anymore. you know why? people who run mailservers for a living got sick of putting up with them and started blacklisting the hell out of them and/or complaining loudly to upstream providers. the administrators of the open relays fall into one of two categories. they either a) noticed that their mailserver didn't work anymore and fixed it or b) didn't notice and the damn thing is now blackholed and/or null routed everywhere. as a result, the spammers decided to not bother abusing them so much anymore. they moved on, and so should you. instead you blow a few paragraphs blathering on and on about open relays and how they retry thier little hearts out.

i'm not even going to start with the faux intelligent tone you wrote this article in. it grates so much that i don't think i could even give a rational fucking criticism.

your research leaves something to be desired. a typical dsl connection can be used to deliver 10k message per minute? really? just how fast is a "typical" dsl connection, anyway? try doing some math on that instead of copying the scariest-sounding statistic you can find on the net. i'm also fuzzy on your claim that "If the message passes all the [SPF and DomainKeys] checks above, we can be sure that our incoming message is not being sent by a known zombie machine, and it is addressed truthfully." how's that work? do you understand the problem that SPF and DomainKeys are trying to solve, or did you just copy that part off of somebody else's site, too? hint: you can register your own domain and point SPF or whatever at *any machine you want*. or how about this: "If you control a mail server, the best thing you can do for yourself and your users is to implement some or all of these systems. The technology is there. It is strikingly effective. It has very few downsides. The ball is in your court." how about false positives? i don't see any mention of false positives in your article. maintenance? you know, software updates, monitoring the system for wedges, reading relevant mailing lists to keep up to date, etc? how about all the time and energy it takes to set things up *correctly*? pretty big downsides. you know what a better idea would be? decomission your vanity mailserver and go use gmail or yahoo or msn or some other system that is run by people who know what the hell they're doing.

i'm feeling grumpy, so i'm going to criticize your measurement criteria, too.

My goal is to stop over 95% of spam that audaciously attempts to lodge itself in my inbox.

hey, i have an idea. shut off your mailserver. i guarantee that you will receive ZERO (that's ZERO, as in a zero with zero zeros after it) spam messages, EVER. the benefit of using my approach is that it takes WAY less time, energy, and money than the approach you used, and it yields better results, at least according to your measurement criteria.

look, here are the problems:

  1. spammers are smart: if you still think that spammers are stupid, please don't even bother replying. i don't want to hear it, you're wrong. they're smart enough to do math, they can automate, many of them can even read. they're perfectly capable of noticing things like HMM MY SPAM CLIENT GETS A LOT OF 4xx ERROR CODES WHEN IT SENDS MAIL, MAYBE I WILL IMPLEMENT A RETRY MECHANISM.
  2. spamming is profitable: machines are cheap. whether you buy them or steal them, you can send a bajillion messages for pennies. reply rates dn't have to be very high to make money on that deal.
  3. they let amateurs like you run mailservers on the public internet

at the end of the day (or the beginning of the day...whenever, really,) this article is just one more in a long line of articles written by fucking amateurs. you didn't provide a single goddamn piece of *basic* advice--that is, advice about the things you need to do to build a strong foundation for your mail handling. here, let me be constructive for a second:

  1. read your logs: and wipe that smirk off your face. you don't have time to go through your logs line by line? well then don't. run them through a parser and pull stats and anomalies. i mean, you are paying attention to what your mailserver is doing, right? if only to figure out which of these spam killing mechanisms are working?
  2. pay attention to the *basic* configuration parameters. welcome to the wonderful world of configurable timeouts, retries, message size limits, limits on number of bcc recipients allowed, etc.
  3. turn on the message sanity checking options. postfix, right? smtpd_recipient_restrictions is the option you're looking for. start turning some of that shit on. for example, reject_unauth_pipelining. you know who uses pipelining? spammers and probably about three legitimate applications. here's a good one: reject_unknown_sender_domain. works pretty well as a low grade sender verification check that, as a bonus, doesn't dos anybody's mailservers. three more options that you should look at include reject_unknown_hostname, reject_invalid_hostname, and reject_unknown_client. those are checks on client ip addrs and helo names. but hey, you're smart, you run your own mailserver, i'm *sure* you've already got all that turned on.
you should be doing *all* of these things before you go messing with spamassassin or greylisting or whatever.

all your points have been made before, and they've been made in better researched, better written articles. feel free to stop with the retreads. if you would like to write an interesting and informative article, how about writing a followup article that talks about how much time and effort is being spent on greylisting by mailserver admins all over the world, all to put up a little speed bump that any halfway intelligent spammer will automate around in a single afternoon. this will happen on a widespread basis in the next year, by the way. anywho, this followup article should then go on to talk about how you are going to modify your system to adapt when greylisting becomes useless.

ok, i'm done now.



Response (none / 0) (#43)
by Cluster on Fri Dec 30, 2005 at 03:52:50 AM EST

i'm not even going to start with the faux intelligent tone you wrote this article in. it grates so much that i don't think i could even give a rational fucking criticism.

For this I apologize.  It is not intentional.  That is simply how I write.

a typical dsl connection can be used to deliver 10k message per minute? really? just how fast is a "typical" dsl connection, anyway? try doing some math on that instead of copying the scariest-sounding statistic you can find on the net.

Yeah, with my own numbers (1.5 Mbit/s, each message is 2,000 bytes) I get less than 100.  I don't know what the author of that page was thinking.  Not verifying this number is an oversight of mine.

hint: you can register your own domain and point SPF or whatever at any machine you want

Yes, but this is intended and desired behavior.  See "SPF doesn't really STOP spam, does it?".  The point is that a spammer will be forced to get his own domain name instead of spoofing his emails with, say, Comcast's.  Once this happens, the paper trail for domain name registration will lead straight to him.

how about false positives? i don't see any mention of false positives in your article.

Well, what can I say... I haven't gotten a false positives yet, and I review my Postfix logs very thoroughly.  In my opinion, the biggest contender for producing false positives is SpamAssassin, since it tries to be the most intelligent of all my mentioned schemes.  Let's go over my schemes again and see where false positives may lie:

  1. RBL: [theoretically] an IP doesn't get blocked unless it spams.
  2. SPF: cannot produce false positives unless it's misconfigured by the DNS administrator.
  3. DK: my mailserver does not verify this on incoming mails; it only signs outgoing mails.  However, this cannot produce false positives unless it is misconfigured by the DNS administrator or sending mail server.
  4. Greylisting: no false positives if the sender follows the RFC.
  5. SpamAssassin: possible.  Separate rather than delete potential spams.
  6. HashCash: only used for lowering your SA score; in the worst case (if generating bad hashes) it's entirely ineffective.
Another reason that false positives are less of a concern for items 1 through 3 is that the mailserver rejects triggering messages immediately, so the sender knows immediately that it bounced and hopefully understands why thanks to the somewhat-informative bounce message.

maintenance? you know, software updates, monitoring the system for wedges, reading relevant mailing lists to keep up to date, etc? how about all the time and energy it takes to set things up correctly? pretty big downsides.

This requires very little additional maintenance; supposedly you already maintain and upgrade Postfix regardless of your anti-spam solutions, and SpamAssassin (again) is going to be the biggest concern, since it's always tweaking and perfecting its algorithms.  HashCash, DomainKeys, Greylisting, and especially SPF are mostly set-and-forget.

you know what a better idea would be? decomission your vanity mailserver and go use gmail or yahoo or msn or some other system that is run by people who know what the hell they're doing.

The fact remains that I get less spam than they.  I guess it's because I don't know what the hell I am doing.  You still have not pointed out anything that I am doing wrong, so this is a non-sequitur.  Thanks for your opinion.

you didn't provide a single goddamn piece of basic advice--that is, advice about the things you need to do to build a strong foundation for your mail handling.

You're right, and that's intentional.  This article is not called "Installing and configuring a mail exchanger".  It is specifically for dealing with spam.  Reading logs is great advice which every system administrator should follow, but I am sure that if I mentioned that, I'd look even more condescending in your eyes.  Condescension is not my goal.

you should be doing all of these things before you go messing with spamassassin or greylisting or whatever.

Indeed.  Set up a well-functioning mailserver using good practices and recommended server options, and then to learn about anti-spam solutions at the server level, read this or similar articles.

[ Parent ]

Don't re-invent the wheel:Still false positives! (none / 1) (#45)
by redelm on Fri Dec 30, 2005 at 10:38:04 AM EST

To me, false positives are the biggest problem in spam fighting. Not the techniques used. The first question you must answer is: how much does leaked spam cost, and how much does a false positive cost? I've seen companies totally disable their systems when an important [purchase] email was false-positived.

Don't re-invent the wheel! There's lots on the 'net about spam fighting by people who are further along than you. You're foundation is the RBL, you might want to look into it here. Blocklists are inheirently flawed because even when well-managed, they're subject deliberate attack to use their DoS services.



[ Parent ]

don't mind him (none / 0) (#50)
by el_guapo on Fri Dec 30, 2005 at 07:33:53 PM EST

(her?) i for on appreciated this. i'm a cisco jockey by training, and i find any 'server' level stuff just IMPOSSIBLE to understand. don't know why he's dogging you, nice article - thank you :)
mas cerveza, por favor mirrors, manifestos, etc.
[ Parent ]
You forgot PostGrey (none / 1) (#52)
by tzanger on Fri Dec 30, 2005 at 11:03:36 PM EST

http://isg.ee.ethz.ch/tools/postgrey/

This has got to be one of the most effective spam/virus blockers I've ever used.  It's simple, it's (very) light on resources, it's auto-tuning and it never misidentifies a message.  

I won't use C/R systems.  Combined with a couple blacklists and some basic sanity checks, this is the bulk of my antispam/virus.  I'm continuously amazed at how effective this works, at least until the spambots include full queueing SMTP servers.

I didn't forget (none / 0) (#56)
by Cluster on Sat Dec 31, 2005 at 11:31:49 PM EST

PostGrey is actually the greylisting policy server that I use, and I covered greylisting pretty thoroughly.

[ Parent ]
How the hell... (3.00 / 2) (#53)
by Armada on Fri Dec 30, 2005 at 11:04:44 PM EST

...did this make it through the edit queue?

All you need to implement from this list is greylisting and perlmx. That's pretty much it. This could have been an article about greylisting and it would have been great, but they had to add in extra crap that doesn't even apply anymore: OPEN RELAYS?!?!?! HELLO!

Anyway, others have cut this article down already, but I find it sad this made it to the front page.

SBL+XBL is still very important (none / 0) (#55)
by Cluster on Sat Dec 31, 2005 at 11:29:31 PM EST

Armada, it's true that greylisting is very effective on its own, but SBL+XBL checking, as offered by Spamhaus, is still very valid and stops a lot of spam before it reaches greylisting.

Here's some proof.

Here's how a typical rejection message looks in the logs:
Dec 31 18:39:21 [postfix/smtpd] NOQUEUE: reject: RCPT from cpe-024-211-049-096.sc.res.rr.com[24.211.49.96]: 554 Service unavailable; Client host [24.211.49.96] blocked using sbl-xbl.spamhaus.org; http://www.spamhaus.org/query/bl?ip=24.211.49.96; from=<EllenGibson@mylifeplus.net> to=<abcdefgh@qnan.org> proto=SMTP helo=<cpe-024-211-049-096.sc.res.rr.com>

Let's search for matches:
pmw@qnan /var/log/mail $ grep -r "sbl-xbl" . | wc -l
262

This is 262 messages in only the last few weeks that were blocked even before beginning to transmit any data.

It's probably true that greylisting would have stopped most of them anyway, but why waste the resources when this is so effective?

[ Parent ]

Is this article from 2003? Things have changed (none / 1) (#54)
by lodc on Sat Dec 31, 2005 at 05:03:34 PM EST

Most of the ideas in this article are old and the techniques have been proven to be less effective than currently available options. If you want to stop spam on your network, here's a quick list of the resources I've used in the past year to obtain excellent results.

If you really want to know the history of spam (and the people who spam):
"Spam Kings" by Brian McWilliams
http://www.oreilly.com/catalog/spamkings/

If you want a guide to using the many settings built in to postfix that can reduce spam:
The Book of Postfix
http://www.nostarch.com/postfix.htm

An excellent book that will both explain why the methods covered in the parent post do not work well (and will continue to become less effective in the future), and show you how to implement modern solutions that are much more effective:
"Ending Spam" by Jonathan A. Zdziarski
http://www.nostarch.com/endingspam.htm

And finally, an open source implementation of modern filtering that integrates easily with postfix (and maintained by the author of the above book):
http://dspam.nuclearelephant.com/

If you have Safari from OReilly, all these books are available online there. They also give a 2 week free trial for anyone wanting to check it out.

PS I am not affiliated with any of the mentioned books, projects, or services... I am just found them all to be excellent.

Happy New Year

-LodC

Good rounded article (none / 0) (#62)
by truesz1 on Tue Aug 01, 2006 at 06:37:17 AM EST

But what about fail-over from DOS spam attack?
Online Backup - Remote Backup - O
Fighting spam at the server level | 62 comments (52 topical, 10 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!