Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
TMDA Ends Spam

By Kyle in Op-Ed
Thu Aug 21, 2003 at 10:49:23 PM EST
Tags: Software (all tags)
Software

Tagged Message Delivery Agent (TMDA) is a challenge/response style anti-spam system which I've been using successfully for about six months. The system is based on a white list. When someone not on the list sends me a message, their message is held in a queue, and a challenge is emailed back to them. When they respond to the message, they're added to my white list, and their original message is delivered to me.

I've seen some resistance to systems like TMDA. This resistance comes in two basic objections: (1) it doesn't work, and (2) it's too rude. These objections are based on encounters with systems with various obvious faults, but condemning all challenge/response systems based on some bad implementations is like saying that cars are a bad idea because the Model A Ford has some problems. I intend to show that TMDA works well and that the real debate is over whether a system like it makes victims out of the people who aren't using it.


Whether challenge/response cuts down on spam is a closed question. I used SpamAssassin for a long time and was very happy with it, but as the flow of spam increased, the flow of false negatives reaching my inbox increased also. When I switched to TMDA, spam stopped dead. Last month I received over 10000 messages. TMDA challenged 1770, and I saw one or two spams.

With that out of the way, lets look at some other non-issues I see raised over and over.

You can't get notices from web sites! You can't do mailing lists! You can't track packages!

I'm on several mailing lists. I get notices from web sites too. With TMDA, when I sign up for these, I can use keyword addresses which pass through without confirmation. If one of them is ever discovered by a spammer, I can dump it. Alternately I can white list the web site's domain.

TMDA is also configured so that when I send a message to one of these lists, it appears to come from the address that I used to subscribe to the list (so I don't have to monkey with it manually in my mailer). For more info, see TMDA FAQ entry 4.8. How do I use TMDA with mailing lists?

Some have complained that challenge/response systems interfere with mailing lists by challenging the list mail that comes in. TMDA specifically tries to avoid that.

E-cards won't work.

I tried sending myself two Yahoo! Greeting cards as a test. In the first test, where Yahoo! asked for the "from" address, I entered an address that is in my white list, and it went through fine. In the second test, I entered a blocked address, and TMDA sent a challenge to Yahoo! (which was ignored). I'm not too concerned about not getting greetings from people I don't know, so this isn't a problem for me.

People using this system have sent me messages, and when I reply to them, I get challenged!

The simplest way to handle this is to automatically white list anyone to whom you send an email. The only problem that has is when you send to a role account (e.g., support@example.com) and get a reply from an individual (e.g., bob_the_tech@example.com). The better solution is to send outgoing mail from a dated address which will pass through TMDA unchallenged until an expiration date. For more detailed information, see the TMDA FAQ item 5.5. When someone replies to my messages, will they get challenged?

Two people using this can't email each other.

This is solved the same way as guaranteeing that when I send a message, the recipient can reply.

Spammers can just set up auto-repliers to get through.

This is covered best in the TMDA FAQ item 1.1. Can't spammers just setup an auto-responder to defeat TMDA? and a little more in 1.13. What do I do when a spammer does confirm their message?

Briefly, if spammers started doing this, there would be two results:

  • The cost of sending spam increases, as spammers have to expend more resources to do it.
  • The spammer now has to supply a valid email address, which makes them easier to track down.

Note that some legislators have tried to force spammers to put valid return addresses on their messages with little effect. TMDA enforces that law in code better than any government.

What about the blind or visually impaired or those who simply fat finger the answer to the challenge?

TMDA uses ordinary email for its challenges, not a challenge image. It doesn't require any functionality outside normal email use. There has been talk of having challenges that require some intelligence to answer, but at the moment it's not necessary.

To fat finger the answer would require some unusual circumstances. The response that's required is to send any message to a unique address, and that address is in the Reply-To header of the challenge. To mess this up, you'd need a mail reader that doesn't honor Reply-To, and you'd have to incorrectly copy the unique address to the mail you try to send.

Spammers forge the address of an innocent victim when they spam, so that bystander will get a huge pile of challenges.

That is unfortunate, but it's no different than:

  • Vacation messages.
  • Bounce messages.
  • List subscription confirmation messages.
  • List moderation notices ("Your message is being held pending moderator approval").

That last sounds the most like the challenges we're talking about. Should mailing lists stop doing that because of the innocents affected? (Incidentally, TMDA is designed not to try to confirm automated messages like the above.)

My point is that given you have to deal with bogus automatic responses generated by spammers already, this is not such a larger problem. The responses generated by TMDA are made to look automatic to existing software. In these terms, a TMDA user is no more a burden to the network than a user who abandoned an address (perhaps because it was being spammed as heavily as the user who switched to TMDA).

Spammers can hijack it to send to third parties.

They can also do that with the other types of automatic responses mentioned previously.

Spammers forge from legit addresses, hoping to hit your white list.

Since virtually every anti-spam system has a white list, this technique is not specific to TMDA. This technique will become more prevalent as all spam filters get better.

No really, the white list is its Achilles heel!

Spammers can already probe a domain (with a dictionary attack) to find legit email addresses. TMDA doesn't change that. Knowing that, they might try to send to those address with forged messages from other addresses in the domain, hoping that everyone in the company has everyone else white listed. A few points:

  • The TMDA FAQ entry 4.10. SPAM is getting in by spoofing my domain discusses solutions to this problem.
  • Probing is harder than harvesting, so the cost of spam goes up.
  • The larger a company is, the more of a problem this can be (for them) if they white list their whole domain. If they don't white list the whole domain, the spammer's job is much harder, and the company email system is only a little harder. A small company is, I expect, too small a target to make this attack worth the spammer's time.
  • It's worth repeating that virtually every other anti-spam system has this problem, but not all of them have a solution. With TMDA, you can take an individual off your white list and give the person a keyword address they can use to email you.

Answering a challenge is too great a burden for innocent senders.

This is covered in TMDA FAQ item 1.5. Won't senders just refuse to confirm their messages?

I think that responding to a challenge is no more of a burden than, say, postage. (Postage has a different purpose, but it is still a small barrier to unlimited communication.)

Some people may choose not to correspond with someone on the basis of their usage of a challenge/response system, and that's their prerogative. In a sense, that's collateral damage. I think it's superior to every other collateral damage I've heard of since the people hurt by it are selecting themselves. (Your message isn't important enough to take a minute to confirm? Fine. Your message is too important to be blocked by a RBL or content filter? Tough.) That being the case, I like losing mail that way a lot more than losing it the way I used to (where the victims don't even know it happened). Heck, if some stranger doesn't think their message is worth another minute of time to get it to me, how much can it be worth to me?

This is "guilty until proven innocent"

I can understand an objection to challenge/response if you think of it as an assertion of guilt. I think of it as an assertion of automation. I'm not trying to verify that you're legally allowed to talk to me; I'm trying to verify that you're a human. In this regard, challenge/response is the same as mangling your email address when you publish it so that it won't be found by spammers' harvesting robots. If I say, "my address is kyle@painted.toehold.com (apply nail polish remover to email me)", I'm not asserting guilt. Additionally, this method blocks email from users who are too dim or inexperienced to decipher these tiny intelligence tests.

But there has to be some down side.

In "Reflections on the 25th Anniversary of Spam," Brad Templeton compares various anti-spam systems. He describes content-based filters this way: "In terms of effectiveness, these are 2nd only to challenge/response tools." The TMDA FAQ lists other systems like TMDA.

  • I found configuration/setup difficult. This is probably because I had installed an old Debian package while referring to current documentation.
  • It increases bandwidth usage, but this is mitigated by combining TMDA with other filters and automatically blacklisting based on messages that never get confirmed (so that it doesn't challenge them again).
  • Sometimes I wonder if challenges get killed by spam filters at the other end, but I have no proof one way or the other.
  • It's a server side solution (like SpamAssassin), but it also has client-side utilities (a CGI script for browsing the pending queue, and an SMTP proxy for sending tagged messages).
  • It's in development and changes frequently. That having been said, it has always worked reliably for me, and it's on its way to stabilizing.
  • I've heard folks say that inexperienced users find the challenges confusing (thinking they're bounce messages). To combat this I've replaced the stock challenge text with one I wrote myself.

The last word

Imagine you email someone you don't know. You get back a magic message, which requests politely two small actions: (1) hit "reply", and (2) hit "send". Performing these actions will allow you to communicate freely and reliably with this person (without being filtered for using the wrong words) and virtually eliminate spam from their mailbox. Do you do it?

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
How do you deal with spam?
o Eat it 21%
o Content-based filters 39%
o Blacklists 2%
o Collaborative filtering 3%
o Spamtrap filtering 5%
o Challenge/response 3%
o My ISP does it 7%
o Hide my address 15%

Votes: 76
Results | Other Polls

Related Links
o Yahoo
o Tagged Message Delivery Agent
o challenge/ response
o some resistance
o SpamAssass in
o keyword addresses
o TMDA FAQ
o 4.8. How do I use TMDA with mailing lists?
o Yahoo! Greeting
o dated address
o 5.5. When someone replies to my messages, will they get challenged?
o TMDA FAQ
o 1.1. Can't spammers just setup an auto-responder to defeat TMDA?
o 1.13. What do I do when a spammer does confirm their message?
o 4.10. SPAM is getting in by spoofing my domain
o keyword address
o 1.5. Won't senders just refuse to confirm their messages?
o Reflection s on the 25th Anniversary of Spam
o other systems like TMDA
o Debian package
o combining TMDA with other filters
o automatica lly blacklisting based on messages that never get confirmed
o on its way to stabilizing.
o filtered for using the wrong words
o Also by Kyle


Display: Sort:
TMDA Ends Spam | 227 comments (200 topical, 27 editorial, 0 hidden)
It's a pain in the ass (4.16 / 12) (#1)
by Demiurge on Wed Aug 20, 2003 at 11:42:40 PM EST

I(and the average e-mail user) don't want to have to spend twenty minutes sending validation e-mails instead of deleting spam.

I(and the average e-mail user) do want to see considerable civil and criminal penalties levelled against serious spammers. Technology can assist in fighting spam, but legislation is needed to really put a stake through its heart. When sending out 50,000 messages on how to enlarge your penis using a forged e-mail address is a felony with jail time attached to it, there will be less spammers.

Responding to challenges vs. deleting spam (5.00 / 1) (#3)
by Kyle on Wed Aug 20, 2003 at 11:55:21 PM EST

Perhaps I'm not the average email user, but I spent far more time dealing with (un)filtered spam than I have spent emailing people I've never emailed before. If everyone on the Internet used TMDA, I'd still be coming out ahead compared to using filters.

[ Parent ]

How many people will return ack. e-mails? (5.00 / 1) (#12)
by Demiurge on Thu Aug 21, 2003 at 03:25:17 AM EST

What about the non-techies who don't? And simply not communicating with them over e-mail isn't an option. Are you going to tell your pointy-haired boss you can't talk to him through e-mail because he's too stupid and lazy to use it?

[ Parent ]
You whitelist him (5.00 / 2) (#21)
by curien on Thu Aug 21, 2003 at 09:31:33 AM EST

Of course, if he changes e-mail addresses (or happens to decide to e-mail you from his Yahoo account while he's on vacation), you've got a problem. OTOH, you can create a custom address which he will exclusively use to e-mail you. Unless he gives out your e-mail address, you're ok.

--
John Ashcroft hates me for my freedom.
[ Parent ]
Exaggerated but true (5.00 / 2) (#5)
by bugmaster on Thu Aug 21, 2003 at 12:04:55 AM EST

I(and the average e-mail user) don't want to have to spend twenty minutes sending validation e-mails instead of deleting spam.
This claim is exaggerated, but it is also very true. The average user will no tolerate any distraction from his immediate goal, no matter how minute. For example, I've had people tell me, "I'll send you that link if you go on AIM, but I won't email it to you because that takes too long".

This means that any system which is not fully automated is doomed to failure. TDMA, Bayesian filters (that need to be trained initially), etc., all require some minute effort from the user -- which he is not prepared to expend. This sounds irrational considering that deleting spam also requires effort; however, most users are so used to spam now that they regard it as a kind of weather phenomenon -- unfortunate, but unavoidable.

Inicidentally, this also means that security of any kind has to be implemented outside of the immediate network, by someone who is a trained sysadmin. Users will never, ever take the time to track down each "Your account is being terminated ! Give us your credit card number now !" message, and they will always set passwords to blank. If the security gets too tough (challenge/response, non-blank, non-dictionary passwords, VPN clients, etc.), the users will simply refuse to use the system altogether.

I have no idea what drives this behavior, but it has been true in my experience. Maybe some resident k5 sociologist can give us the answer...
>|<*:=
[ Parent ]

I hate to whip out the car analogy, (5.00 / 1) (#18)
by la princesa on Thu Aug 21, 2003 at 08:23:54 AM EST

but it's the same kind of deal with computers.  We live in a society even more specialised than the wildest fancies of our ancestors, and when computery types expect people to do the equivalent of turning their own rotors, they are failing to comprehend that.  Which makes them as clueless as they paint 'average users' to be.  

___
<qpt> Disprove people? <qpt> What happens when you disprove them? Do they disappear in a flash of logic?
[ Parent ]
Cars (5.00 / 1) (#35)
by bugmaster on Thu Aug 21, 2003 at 01:30:16 PM EST

I fully agree -- I am nearly completely clueless when it comes to cars. While I may not have a completely mystical outlook as far as cars are concerned (i.e., I understand that cars are powered by physical processes that are well understood, and can be learned even by me), I still defer car maintenance to my mechanic (even though, come to think of it, I can actually change the oil). I have no problem with that. However, by "defer" I mean just that. I don't try to turbo-charge my engine, and the mechanic does not turbo-charge my credit card. In other words, my car is locked down just like a business computer system should be.
>|<*:=
[ Parent ]
What is? (5.00 / 1) (#65)
by greenrd on Thu Aug 21, 2003 at 06:10:13 PM EST

We live in a society even more specialised than the wildest fancies of our ancestors, and when computery types expect people to do the equivalent of turning their own rotors, they are failing to comprehend that.

What specifically do you think is analogous to "turning their own rotors"? Replying to a TDMA confirmation request email? Choosing a password?


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

How many confirmation emails and how (3.00 / 2) (#95)
by la princesa on Fri Aug 22, 2003 at 12:07:13 AM EST

many passwords is the 'average user' going to be dealing with?  Even bad old majordomo auto-emails often don't require the user to do anything other than click reply.  Just reading the first few paragraphs of this article indicates that the user would have to do more than that for many emails.  And though 'just add domains of whatever you want to whitelist' is presented as easy enough to do, at that point, one's already reached the car equivalent of 'you don't even have to go to the brake shop, you just need to take off your tires and turn your rotors'.  

Turning one's rotors isn't all that hard, if one already has the relevant tools and enough space to do so in, AND THE RELEVANT DOMAIN KNOWLEDGE.  This article breezes over it, but a certain amount of specific domain knowledge (pun un-intended) is required for the whole system to work, and it is domain knowledge one cannot and should not reasonably expect every average user to possess.  Just the concept of a whitelist alone would irk a lot of people who know that they can more easily highlight all and click delete in a whopping 2 seconds instead.  

It is odd that outside of computery things, I've yet to encounter people in other realms of knowledge expecting users to carry a certain bulk amount of domain knowledge just to do simple tasks and blaming the users for not choosing to acquire that domain knowledge.  

___
<qpt> Disprove people? <qpt> What happens when you disprove them? Do they disappear in a flash of logic?
[ Parent ]

Who's the user? (5.00 / 2) (#113)
by Kyle on Fri Aug 22, 2003 at 10:54:05 AM EST

If you're sending to a TMDA-protected address, "just click reply" is really all you need to know. If the TMDA user emailed you, and you replied, you would not be challenged.

If there were ever a problem with you being white listed (i.e., a spammer is pretending to be you), the TMDA-protected user would do the work of generating a new address, and you'd only have to paste it into your user friendly address book (just as if your friend had changed addresses). The worst you could say about it is that the address would be hard to memorize.

The only time you should expect to face a challenge is if you're emailing someone who uses TMDA from an account they've never seen (either because they don't know you or because you moved to a new address since they saw you last).

As for being the user who is protected by TMDA, it can require some know-how. There are tools to make this easier (and even a service you can hire to do the work for you), but given that the software is not up to its first "real" release, the observation that it's not "ready for the masses" should hardly be a surprise.

[ Parent ]

I have (none / 0) (#137)
by greenrd on Fri Aug 22, 2003 at 03:08:04 PM EST

It is odd that outside of computery things, I've yet to encounter people in other realms of knowledge expecting users to carry a certain bulk amount of domain knowledge just to do simple tasks and blaming the users for not choosing to acquire that domain knowledge.

Hmm... let's see. Cooking. Cleaning.

Driving a car is probably the best example.


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

Um, no. (none / 0) (#141)
by la princesa on Fri Aug 22, 2003 at 06:18:27 PM EST

The original fixing cars analogies are more valid in this instance than the mere act of driving a car.  

___
<qpt> Disprove people? <qpt> What happens when you disprove them? Do they disappear in a flash of logic?
[ Parent ]
Interesting (5.00 / 2) (#30)
by Kyle on Thu Aug 21, 2003 at 12:08:59 PM EST

So, if I understand this correctly, the way to get challenge/response accepted is just to force it down the throats of the sheep. Once they are forced to do it for a while, they'll get used to it, it'll be part of their daily noise, and it will be accepted.

This reminds me of this comment posted on an earlier incarnation of this story.

[ Parent ]

Yes. (5.00 / 1) (#42)
by CanSpice on Thu Aug 21, 2003 at 03:23:52 PM EST

Because remember, we know technology and they don't, so we know the best solution to everything, and we must cram it down their throats whether they like it or not. After all, they are just clueless users.

@_@

[ Parent ]

Tried it before (3.57 / 7) (#8)
by buck on Thu Aug 21, 2003 at 01:01:24 AM EST

It's actually pretty good at what it does. My biggest problem with it was when I used it with fetchmail since my incoming email is held in an IMAP inbox at the ISP. Here's the sequence of events:

  1. Fetchmail is first to run retrieving my waiting emails from the ISP passing it off to my local mailer (Postfix). It rewrites the email envelope changing the To address from buck@isp.com to buck@localhost.localdomain (not a registered domain name, obviously).
  2. Postfix seeing the To address as being local, checks for a ~/.forward file which happens to call TMDA, and upon finding it passes the message to it
  3. If the From address is not in the whitelist, TMDA, as mentioned in the article, puts the email in a waiting queue and sends a confirmation back to the sender. This is where you see the results of fetchmail's envelope rewriting cock things up. Here's an example confirmation:
    Subject: Please confirm your message


    Your e-mail message with the subject of "Postfix virtual mappings" is being held because your address was not recognized.
    To release your message for delivery, please send an empty message to the following address, or use your mailer's "Reply" feature.

    buck+confirm+accept.1013266802.14693.049c30@localhost.localdomain

    This confirmation verifies that your message is legitimate and not junk-mail.

    [ This notice was generated by TMDA v0.44 (http://tmda.sf.net/), an automated junk-mail reduction system. ]
    There you have it. The address it gives you is not the one at the ISP, but the one on my local machine. As a result, any confirmation reply will never arrive.

TMDA definitely works as advertised. However, in the way I'd have to use it, it doesn't give me anything that couldn't be done with other filtering utilities.


-----
“You, on the other hand, just spew forth your mental phlegmwads all over the place and don't have the goddamned courtesy to throw us a tissue afterwards.” -- kitten
i heard LRCT is good too... (4.40 / 5) (#9)
by rmg on Thu Aug 21, 2003 at 01:04:19 AM EST

but i notice you did not mention it in the article.

i think it is important to address competing technologies when dealing with a topic like this one. i think you should try to get that in there while it's in editing.

i mean, you don't need to say much, just a few words of acknowledgement, or even a small comparison if you're feeling ambitious.

otherwise good article. +1 FP.

_____ intellectual tiddlywinks

Mixing? (4.33 / 3) (#10)
by jjayson on Thu Aug 21, 2003 at 02:34:03 AM EST

Is there a way to mix systems? I would like to use filtering software, like Spam Assassin or my own, and anything not marked as spam should just pass though. However, whatever the filtering software marks as spam should then be sent to the TDMA authenticating system.

I would like to use Spam Assassin, but I am too scared of losing messages that are not spam (I've had a number of recruiter messages never make it to me from filtering so I had to write my own). I'm willing to get a little spam in exchange for not losing any messages.

This would seem to be a good balance: if the message doesn't look like spam, then it shouldn't be treated like spam. However, if the message does look like spam it should have a higher burden.
--
This space for rent.

Sure is! (none / 0) (#26)
by mge on Thu Aug 21, 2003 at 10:51:51 AM EST

Check out the FAQ: http://tmda.net/faq.cgi?req=show&file=faq01.010.htp

They descirbe what you want, but at a RBL to the mix also.

[ Parent ]

That's not i (none / 0) (#38)
by jjayson on Thu Aug 21, 2003 at 03:05:27 PM EST

They is for people that like to really armor up their email address and want to throw out email with other program, then use the challenge mechanism for whatever is left over. That is, they want to make it ever more strict.

However, I want to use TMDA to be more lenient. Instead of everything that passes Spam Assassin being sent to TDMA, I want everything that fails Spam Assassin to go through TDMA and everything that paases Spam Assassin should just go right to me.
--
This space for rent.
[ Parent ]

Either way (none / 0) (#39)
by Kyle on Thu Aug 21, 2003 at 03:18:07 PM EST

TMDA is typically run from procmail, as is SpamAssassin. You can integrate them practically any way you want, though getting them to use the same white list (they both have them) might take a little scripting.

[ Parent ]

You're a programmer (none / 0) (#66)
by greenrd on Thu Aug 21, 2003 at 06:12:29 PM EST

You know this is possible, easily.

Have you been renting your account out to other people again?


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

programmer, yes (none / 0) (#73)
by jjayson on Thu Aug 21, 2003 at 06:48:24 PM EST

Sometimes even a CS researcher -- but an admin, no.

I don't like messing with my machine. I like it to just work. Read a simple manpage, give it a few options, and be done with it. I'm not the kind of person that likes doing all that administrivia stuff.
--
This space for rent.
[ Parent ]

It' already been done (none / 0) (#127)
by ssentinel on Fri Aug 22, 2003 at 12:41:30 PM EST

TMDA can be configured to run against either the positives or the negative (or both) results from other mail filtering utilities (be that SpamAssastin, Bayesian Filters, or RBLs).

[ Parent ]
Busy, busy, busy world (3.00 / 5) (#11)
by SwampGas on Thu Aug 21, 2003 at 03:12:32 AM EST

People cannot be bothered with that in this busy busy busy world nor do they care about technology.

They forget that they're the same people who used the challenge/response method to gain access to a BBS (remember how it would call you back and you had to pickup the line manually?).  Oh wait...they weren't those people.

We're due for another split soon.  Right now the computer users are all thrown in the mix together.  Sooner or later we nerds shall emerge again using great technology, including this challenge/response email system (probably deprecating the current RFC on SMTP), and we will be happy once again...leaving behind the ignorant soccer moms and kiddies to deal with the hundreds of penis enlargement and toner spams per day :)

Simple (4.50 / 2) (#71)
by greenrd on Thu Aug 21, 2003 at 06:30:50 PM EST

If you can't even be bothered to hit reply, can I be bothered to read your email? I have a lot of things I want to read every day - why should I read your email as well?

In fact I might start using TMDA and put that in my challenge message: "If you can't be bothered to hit reply, then I can't be bothered to read your message."


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

Here Here!! (none / 0) (#125)
by ssentinel on Fri Aug 22, 2003 at 12:37:29 PM EST



[ Parent ]
Baysian filtering works better (4.62 / 8) (#13)
by dirtydingus on Thu Aug 21, 2003 at 04:08:19 AM EST

I used to use popfile. Now I just use Mozilla 1.4's built in filtering. But there are plenty of other choices.

There are two problms with the Challenge response system. The first is the spoofed sender problem mentioned elsewhere. An email that purports to come to you from a trusted friend wll go through, which in particular means that email Viruses do not get stopped by this system. Now you personally may not use windows or outlook expose but most people do and a system such as this will tend to lull them into a false sense of security

Secondly the challenge system works as a great way to harvest addresses. Sending a spam to 10000 different names at a domain and seeing who sends back a challenge enas you have identified a number of legit users in that domain. Then a spam sent from ONE of the legit users to all the others is likely to pas through without trouble since it seems likely that users will tend to whitelist email that purports to come from their own domain (if not you end up getting nastygrams from PHBs).

The obvious advantage of CR vs Baysian filterig is if you are paying per-bit for bandwidth to download the mesasges. I get on average about 50% spam by message count, but spams are usually (viruses excepted) small messages of <5k in size so receiving them and junking them doesn't take long, hence its not an isue for me, however I could see how it might be an issue for people who either pay per bit for spam and get a lot of it or who have slow connections ot their email servers.

DD
People can be put into 10 groups: Those that understand binary and those that don't.

You can already do this... (4.00 / 2) (#25)
by mge on Thu Aug 21, 2003 at 10:34:56 AM EST

Secondly the challenge system works as a great way to harvest addresses. Sending a spam to 10000 different names at a domain and seeing who sends back a challenge enas you have identified a number of legit users in that domain.

Don't most MTA's already send an error message if you send mail to a user at a domain, and they don't exist?

Couldn't you simply get live addresses that way?

Also, as pointed out in the article, for this to work, they'd have to give a real reply-to e-mail address, and now you've got some help in tracking them down.

[ Parent ]

Two points (4.00 / 1) (#44)
by leviramsey on Thu Aug 21, 2003 at 03:34:13 PM EST

An account running TMDA is basically guaranteed to be a real account that people will be communicating with. All the absence of a bounce tells you is that mail [may] be accepted by that address.

Set the Reply-to to a webmail address; the challenges and the bounces will be faster than any abuse complaints which could shut down the account. Alternatively, if you're utilizing trojan'd boxes to be relays (which I suspect is becoming quite common), simply have the trojan'd boxes listen on port 25 for bounces and challenges and report back the results. It would not surprise me if spammers add that to the repertoire, if they haven't already.



[ Parent ]
Is this where you're headed? (3.00 / 1) (#53)
by mge on Thu Aug 21, 2003 at 04:43:36 PM EST

So lets say that a spammer used this to get my TMDA address, spams me, and then cofirms his message (this is a lot of work, but it could be automated if the returns were high enough).

What can I do now?  If he wants to spam me again, he'll either use the same e-mail (in which case I can blacklist the address) or repeat the process again.

In the latter case, TMDA wouldn't block that spam, but TMDA isn't trying to block all spam, just most of it.

[ Parent ]

No (5.00 / 1) (#57)
by leviramsey on Thu Aug 21, 2003 at 04:55:02 PM EST

Read the first post. Once you have a list of valid and active addresses, you don't necessarily spam those addresses, but you use the fact that, whatever anti-spam measures your coworkers are taking, they've likely whitelisted you (or are not going to blacklist you).



[ Parent ]
Oh, Okay. I understand [nt] (3.00 / 1) (#59)
by mge on Thu Aug 21, 2003 at 05:01:49 PM EST



[ Parent ]
Bayesian with > 2 buckets (2.00 / 1) (#93)
by fencepost on Thu Aug 21, 2003 at 11:32:56 PM EST

I'm using POPFile, and find that one of the things that really helps accuracy is having buckets for the different "clusters" of message types I get. Items from mailing lists, for example, tend to have quite a bit in common with each other; similarly with receipts for items I've purchased online; and similarly with spam messages (though I gave up on trying to classify them into sex/finance/drugs/other).

By breaking things into multiple categories, the accuracy of the system as a whole.  For my last 683 messages, I've had >99% accuracy, with 3 messages being misclassified as spam (one was a virus, the other two were from mailing lists) and two false negatives (spam slipping through).  That's since mid-day on Monday, though I've been using this set of filters for ~5 months now.

--
"...when it appears, it is always equipped for the seeker's needs. Dobby has used it, sir," said the elf, dropping his voice and looking guilty, "when
[ Parent ]

Requires normal spam classification too (5.00 / 2) (#14)
by BenJackson on Thu Aug 21, 2003 at 04:39:00 AM EST

If you get a lot of spam (and I do!) your mail queue will become totally unmanagable if you generate one mail per spam. When I bounce spam (using EXITCODE=67 in procmail after spamassassin rules against it) I often get myself unsubscribed from many quasi-legitimate lists, but at the same time the undeliverable bounces turn into a giant mail queue and a flood of postmaster notifications.

So you need a regular filtering system in front of challenge-response to keep things under control. At that point you really are focusing the penalty on the messages least likely to be spam. I really dislike losing nonspam messages, and as long as I'm going to have to deal with spam anyway I at least want to make it as easy as possible for real people to contact me.

It's a FAQ. (4.00 / 2) (#29)
by Kyle on Thu Aug 21, 2003 at 11:59:06 AM EST

4.7. How do I prevent my postmaster from getting all my bounced confirmation requests?

Basically, a config option lets you decide where the bounces will go, so you can send them to some address that will eat them without comment.

[ Parent ]

When I get one of those confirmation message... (4.00 / 8) (#15)
by CtrlBR on Thu Aug 21, 2003 at 04:40:42 AM EST

...I just ignore it.

The fucker assume that I'm a spammer, fine with me. End of the exchange.

And that system has so much DoS potential, if only because of the generated traffic. You can mailbomb a poor soul mailing a bunch of TMDA using people using his address as source, or you can mailbomb a TMDA using person using fake addresses so he not only have the download expense but he has to spend bandwidth uploading confirmation messages too. If you send really short messages you can make him spend way more bandwidth than you used, since the confirmation message is fairly big.

If no-one thinks you're a freedom fighter than you're probably not a terrorist.
-- Gully Foyle

What did you want? (5.00 / 1) (#24)
by mge on Thu Aug 21, 2003 at 10:31:52 AM EST

Most of the time, when I e-mail someone, I want to get a response.  If you send someone a message whom you've never talked to before and you get back a please comfirm yourslef message, and decide to ignore it what do you do after that?

Do you have some examples of who you were trying to e-mail, why, and what you had to since you couldn't get in touch with them by e-mail?

[ Parent ]

Usually... (none / 0) (#28)
by CtrlBR on Thu Aug 21, 2003 at 11:51:38 AM EST

It's not a friend, otherwise I would be white-listed.

Every time it happens to me it's taking in private something from a mailing list or Usenet. I'm not usually motivated enough to bother replying to the confirmation message, I don't think it's worth it...

The guilty until proved innocent part is quite repulsing to me...

So this system effectively make its users lose legitimate mail from people offended or annoyed by it.

If no-one thinks you're a freedom fighter than you're probably not a terrorist.
-- Gully Foyle

[ Parent ]
Not really guilty until proved innocent (5.00 / 1) (#33)
by vadim on Thu Aug 21, 2003 at 12:53:43 PM EST

TMDA just does plain normal authentication. It doesn't ask you for a cryptographic certificate that is signed by a CA that certifies that you don't spam, or anything like that. All it asks to prove is that you're human.

Nobody seems to be offended when Windows or Linux asks you for your password to prove that you're indeed who you say you are. So why are you?
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

I see (5.00 / 1) (#37)
by mge on Thu Aug 21, 2003 at 02:30:31 PM EST

I wonder however, if you really cared about the matter at hand, would you reply to white list yourself?

If so, then the system has a nice side effect, if you don't care enough about the message to hit reply, and then send, maybe I shouldn't either?

[ Parent ]

Replying to challenge... (3.00 / 1) (#58)
by CtrlBR on Thu Aug 21, 2003 at 04:55:48 PM EST

...is supporting the use of the practice of challenge response.

As I am against it because I think filtering on content (I use bmf and spamassassin and seldom get spam nor false positive) is the right way and giving in in replying to challenge response is encouraging a practice that has potential to make everyone life worse.

Better kill that ugly bird while in the egg, or at worst in the nest...

So I don't reply to challenge-response request on ideological ground...

If no-one thinks you're a freedom fighter than you're probably not a terrorist.
-- Gully Foyle

[ Parent ]
A valid reason (I guess) (5.00 / 1) (#60)
by mge on Thu Aug 21, 2003 at 05:04:07 PM EST

I think it a little odd for you to force your views on others, but then again, if you weren't open to other views, perhaps a lot of people don't wish to talk to you?

At least you'll apply the rule consistantly (I hope), otherwise you're just being stuck up.

[ Parent ]

And I think the other solutions don't work too. (5.00 / 2) (#88)
by Kyle on Thu Aug 21, 2003 at 10:04:40 PM EST

I wound up on a RBL once. Those guys create collateral damage with pride.

Content-based filters basically keep you from talking about spam or any other topic that spammers like to talk about. It's almost like censorship.

I try to convince the users of these systems of the problems they have, if they're interested in listening. Ignoring them sure doesn't teach them anything.

[ Parent ]

Stupid (keyword based) content filtering... (none / 0) (#105)
by CtrlBR on Fri Aug 22, 2003 at 05:35:00 AM EST

...keeps you from talking about spam.

Bayesian filtering does not.

If no-one thinks you're a freedom fighter than you're probably not a terrorist.
-- Gully Foyle

[ Parent ]
The penis rule (2.00 / 1) (#19)
by speek on Thu Aug 21, 2003 at 09:23:53 AM EST

The penis rule filters out about 50% of all spam I get. Any email with "penis" in it is gone. Other rules I've thought about implementing include, the "mortgage" rule, and the "html formatted" rule.

--
al queda is kicking themsleves for not knowing about the levees

You fool (4.42 / 7) (#22)
by Big Dogs Cock on Thu Aug 21, 2003 at 09:59:18 AM EST

I have answered them all. My penis is now 43 feet long.

People say that anal sex is unhealthy. Well it cured my hiccups.
[ Parent ]
That's all? (none / 0) (#32)
by Kyle on Thu Aug 21, 2003 at 12:50:44 PM EST

That's only 516 inches. You could have had 50,000 inches with only a little spamming.

[ Parent ]

spam misspellings (4.50 / 2) (#48)
by eudas on Thu Aug 21, 2003 at 03:48:09 PM EST

yeah but then they start moving to pen1s, p * e * n * i * s, and all other kinds of permutations.

lots of people can't (or won't, for the reason above) spell mortgage; you get alot of morgage mail as well.

etc etc .. then you wind up with a million rules for a million spams. feh.

eudas
"We're placing this wood in your ass for the good of the world" -- mrgoat
[ Parent ]

if it were a huge problem (4.33 / 3) (#62)
by speek on Thu Aug 21, 2003 at 05:18:35 PM EST

I could always outsource my mail filtering to some guy in India. Probably cost less than some fancy software package.

--
al queda is kicking themsleves for not knowing about the levees
[ Parent ]

Costs Less (4.00 / 2) (#123)
by ssentinel on Fri Aug 22, 2003 at 12:24:01 PM EST

TMDA is Free, Runs on your mail Server (not the client) and works great

[ Parent ]
Er. (5.00 / 1) (#68)
by awgsilyari on Thu Aug 21, 2003 at 06:25:42 PM EST

yeah but then they start moving to pen1s, p * e * n * i * s, and all other kinds of permutations.

And your trusty Bayesian filter just adds those to its list of spam words, problem eliminated.

Wait, you don't run an automated Bayesian filter? For shame.

--------
Please direct SPAM to john@neuralnw.com
[ Parent ]

bayesian filters (3.00 / 2) (#94)
by eudas on Thu Aug 21, 2003 at 11:57:03 PM EST

i suppose i should do some research into what exactly a bayesian filter is, and how it works..

eudas
"We're placing this wood in your ass for the good of the world" -- mrgoat
[ Parent ]

Why we need SPAM filtering (2.66 / 3) (#20)
by lorcha on Thu Aug 21, 2003 at 09:28:01 AM EST

I'm sure it was only a coincidence, but today's Dilbert comic shows what happens without TMDA...

;)

--
צדק--אין ערבים, אין פיגועים

I've only hit TMDA once (4.42 / 7) (#23)
by mge on Thu Aug 21, 2003 at 10:28:44 AM EST

There's only been one person whom I've e-mailed that was running a TMDA system.  I got his e-mail address off a mailing list after searching google.

I'd much rather have to hit [reply] and have my message get to him, then have it lost with a bunch of spam (or worse, find that he no longer uses that e-mail address!)

TMDA and it's ilk are a great system, I wasn't offended at all when I was asked to reply, I'm sure if you wanted something, you wouldn't be either.

Nice of you (3.16 / 6) (#41)
by leviramsey on Thu Aug 21, 2003 at 03:20:53 PM EST

To encourage a broken design. I have yet to encounter a TMDA challenge, but if I do, I doubt I'll reply. If one is stupid enough to use TMDA, I'm not inclined to communicate with them, unless they've pre-established their worthiness to me.



[ Parent ]
Exactly. (4.00 / 2) (#52)
by mge on Thu Aug 21, 2003 at 04:32:12 PM EST

I'm not inclined to communicate with them, unless they've pre-established their worthiness to me.
They had.  They had solved a problem I was having, and posted information about it to a mailing-list.  I wanted some more information, so I emailed them.

I wanted information, and they had "pre-established their worthiness" by showing they had that information.

How exactly am I encouraging a broken design?  I want information, they want to know I'm human.

At least this way I'm more sure my message gets to the user.  I can never know if I'm on some blacklist they've built, or my message is tagged as spam and sent to the bit-bucket.

[ Parent ]

I just noticed this (4.40 / 5) (#55)
by mge on Thu Aug 21, 2003 at 04:49:54 PM EST

When replying to another comment of yours, so forgive the extra posting.

You list your e-mail address as:

lramsey.student@umass.edu (swap @,.)

That's a mail-to link.  When I click it, my e-mail client opens up with your address in the from box.  I now have to do two things

  1. Delete the (swap @,.) thing
  2. Preform the swap
Step two actully takes a few more clicks, but that is because my e-mail client here is stupid (Microsoft Outlook) and sees your address a valid email addres (in form) and forces me to double click on it so I may edit it.

I had to do all this work, just to prove to you that I'm not a bot harvesting addresses.

I would have much rather had you running TMDA, put in your real e-mail address and be forced to  hit reply and then send ONCE, instead of this "decode the e-mail" address process every time I want to send you a message.

[ Parent ]

Innocuous challenge (4.00 / 2) (#77)
by phliar on Thu Aug 21, 2003 at 07:48:50 PM EST

I have yet to encounter a TMDA challenge, but if I do, I doubt I'll reply.
What if the challenge looked something like this:
Hi, I see you sent me email earlier today. I'm having some spam-related email problems -- could you re-send your message? Thanks!
Of course this has the small drawback that the message body is re-sent even though it's not required. The only other thing left to do is: write a little script or something to look at your "sent messages" folder and find all the addresses you've sent email to; put that in your whitelist. Hmmm... that actually sounds good, I think I'm going to try that and see how I (and others) like it.

Faster, faster, until the thrill of...
[ Parent ]

TDMA (4.50 / 4) (#107)
by gruk on Fri Aug 22, 2003 at 06:53:01 AM EST

I place TDMA-challenges in exactly the same bucket as I do with munged mail addresses. If the person I am trying to reply to makes me jump through hoops to reply, I won't. End of story.

Possibly a bit harsh, but taht's the reality I come from. If s[pam bothers you so much, perhaps *not* having an email address is the way to go?

[ Parent ]

TMDA (3.50 / 2) (#121)
by ssentinel on Fri Aug 22, 2003 at 12:19:28 PM EST

Properly configured, TMDA won't make you jump through a hoop if you are replying to a message that I sent to you in the first place.

On the other hand if you are trying to contact me without any prior relationship having been established, you will be challenged.

Possibly a bit harsh, but that's the reality I come from. If challenges bother you so much perhaps not e-mailing me is the way to go.

[ Parent ]

Missed opportunities (4.50 / 2) (#200)
by srivasta on Mon Aug 25, 2003 at 01:02:38 PM EST

Most of the time I have been hit with CR challenges while trying to answer questions or sending solutions on the Debian user mailing list; and I have no intention of responding to a system that assumes mail from me is spam ("mail shall not be read unless ..."). I have already done my good deed for the day by trying to be helpful.

        Of course, if you do not want to hear my response, it is no skin off my nose, and it is your decision (and quite possibly, your loss).

        Through the last couple of decades, both in my business life, and my personal one, I've had unexpected emails whose content was important (and sometimes critical) to me. Prospective clients have come out of the blue, as have high school friends from 25 years ago. I would not be boorish enough to tell them that by default I consider them spammers; indeed, I would had been the loser in many of these opportunities.

        Of course, one can define away any potential problem by calling it a feature: You could equally say that any email not coming from gorgonzola.att.net is spam, and delete them, you still would have zero spam. (and no email, but hey, we defined that not to be a problem).

        Any mechanism that eliminates spam is likely to have false positives and false negatives; challenge response systems get a low false negative rate at the expense of a far higher false positive rate; which some people (including me) find unacceptable.

        For the record, I treat challenges like I do all my unsolicited email: send it to the bit bucket.

[ Parent ]

C-R Systems Assume That Everything Is Spam (4.40 / 5) (#31)
by dasunt on Thu Aug 21, 2003 at 12:36:57 PM EST

The problem with Challenge-Response systems is that they assume all unknown email is spam.

There is an opt-out clause for the sender, but, in practice, what percentage of challenges to legitimate mail receives no response?

I would like to see the percentage of false positives sent to the great bit bucket in the sky for C-R systems in the real world.

[And I'm still not comfortable with assuming that everyone who emails me is trying to spam me. I'll complain when the companies assume that I'm a pirate and put anti-copying mechanisms on game disks, and I'll complain when I get a challenge for a legitimate email.]

However, I do see a use for C-R systems. Currently, I have procmail setup to presort some mailing lists to certain folders, then it runs everything else through bogofilter, sorting any potentian spam into a 'Maybe Spam' folder. When I check my email, I tend to scan my spam folder and look for any noticeable misclassifications. It might be worth it to set up a C-R to any emails in my spam folder that goes unread.

OTOH, with some forged headers and an open relay, it would be easy to flood some email address with challenges from a fake address. The fake address gets flooded with challenges (which would not be whitelisted), but the fake address would reply to each challenge with its own challenge (which would be whitelisted). If each spam would do this, the system would be flooded with illegitimate challenges. In the long run, C-R might make spam unprofitable. In the transitional period, spam might destroy C-R.



Challenge-Response systems (2.00 / 1) (#40)
by ti dave on Thu Aug 21, 2003 at 03:20:09 PM EST

When you're at war, it works pretty damn well.

False positive results can be dealt with.

I'm almost drunk enough to go on IRC. ~Herring
[ Parent ]

Guilt vs innocence. (4.42 / 7) (#50)
by Fredrick Doulton on Thu Aug 21, 2003 at 04:06:31 PM EST

'This is "guilty until proven innocent"'

No, this is "I don't want you coming into my home unless I know who you are."

Bush/Cheney 2004! - "Because we've still got more people to kill"

I like it (4.00 / 4) (#51)
by pyramid termite on Thu Aug 21, 2003 at 04:28:52 PM EST

It sounds simple and clear to use; and as far as people being offended that you've asked them to verify their spam-free status, well, if they're that uptight about having to hit reply and send again, they're probably not fun people to know anyway.

On the Internet, anyone can accuse you of being a dog.
Massive procmail file (3.33 / 3) (#54)
by BadDoggie on Thu Aug 21, 2003 at 04:46:27 PM EST

I write my own filters, you insensitive clod!

Not that I have anything against all the other methods; I'm just too lazy to implement some.

More importantly, I want the spam dead at the mail server, not at the client.

woof.

"Non videri sed esse." — Tycho Brahe "Not to be seen but to be."

TMDA runs at the mail server. (4.00 / 2) (#86)
by Kyle on Thu Aug 21, 2003 at 09:51:28 PM EST

If it makes any difference to you, TMDA is meant to run on the server. It has some nice tools for off-server clients to access its features, but it would be difficult to run it purely on a client system.

[ Parent ]

I refuse to answer any TMDA-type mail confirmation (3.60 / 10) (#61)
by cbraga on Thu Aug 21, 2003 at 05:11:02 PM EST

If the lazy addressee wants me to do the work, instead of deleting his spam or getting a decent workable filter, I say fuck him.

ESC[78;89;13p ESC[110;121;13p
um...lemmie guess, (2.50 / 2) (#63)
by Run4YourLives on Thu Aug 21, 2003 at 05:19:12 PM EST

You don't bother answering the door when someone knocks either.

Let me take another one, you don't have many friends, do you?

I'm not implying that not having friends is bad, just giving you something to consider...

It's slightly Japanese, but without all of that fanatical devotion to the workplace. - CheeseburgerBrown
[ Parent ]

Wrong analogy... (4.25 / 4) (#67)
by loucura on Thu Aug 21, 2003 at 06:16:38 PM EST

Challenge/Response email filtration is more along the lines of calling someone (say your mother), and then getting a call back saying that if you want to talk to that person, you have to call back. It's not the senders responsibility to help keep down the spam of the recipient. That said, I'm not going to do extra work just so you can avoid getting a couple unsolicited commercial emails.

[ Parent ]
Kumbaya! (3.00 / 1) (#70)
by Kyle on Thu Aug 21, 2003 at 06:30:34 PM EST

It's not the senders responsibility to help keep down the spam of the recipient.

Man, whatever happened to "the bandwidth you save will be your own"? Not that my disillusionment is your responsibility or anything.

[ Parent ]

Phone analogies are flawed (2.00 / 1) (#124)
by phliar on Fri Aug 22, 2003 at 12:28:23 PM EST

Email is not like a phone call. If you want to use analogies, it's more like postal mail or phone tag. Here's the analogy for you to critique:
Challenge/Response email filtration is more along the lines of sending a letter to someone (say your mother), and then getting a letter back saying that if you want to correspond with that person, you have to send another letter back.
Seems the people arguing about this have different expectations from email. Is the sender doing the recipient a favour by sending the message, so "fuck him if he wants me to jump through hoops!" or is the recipient doing the favour so "if I want my question answered, I'll have to reply to this challenge."

I'm assuming that only strangers (strange addresses, rather) get challenges, all your friends (and mothers) are already on your whitelist.

(On the phone thing: sometimes when a friend (or mother) calls me I don't recognise the voice. My frostiest dealing-with-telemarketers-voice gets switched in as I say "Who is this?" Maybe my friends (and mothers) are unusually thick-skinned, but I've never had anyone get annoyed that I didn't recognise their voice.)

Faster, faster, until the thrill of...
[ Parent ]

I hadn't thought of that. (3.00 / 1) (#130)
by Kyle on Fri Aug 22, 2003 at 01:05:04 PM EST

Seems the people arguing about this have different expectations from email. Is the sender doing the recipient a favour by sending the message, so "fuck him if he wants me to jump through hoops!" or is the recipient doing the favour so "if I want my question answered, I'll have to reply to this challenge."

This is a good insight. I definitely look at it the latter way. I don't feel obligated to read whatever is sent to me. If you're sending to me, I view that as requesting the favor of reading the message.

[ Parent ]

Drawbacks of C-R (4.00 / 1) (#134)
by phliar on Fri Aug 22, 2003 at 02:12:18 PM EST

After having thought about this for a couple of days now, I can think of one case where the challenge may be obtrusive (or offensive):
Hey, I liked Unicon so much I want to give you $1000. Send me your USMail address or paypal account if you have one.
Alas, I haven't received a message like that. (Yet!)

Ok, I'm convinced. Before I go full-bore, though, I'm going to make TMDA the recipient of messages flagged as spam, and see how my correspondents feel.

Faster, faster, until the thrill of...
[ Parent ]

Starting with TMDA. (3.00 / 1) (#135)
by Kyle on Fri Aug 22, 2003 at 02:22:23 PM EST

For the first month that I used it, I paid close attention to the pending queue to see if it had blocked a web site mailing that I wanted. I've also seen it suggested that you configure it so that it holds filtered mail instead of confirming. Then you can watch the pending queue closely and iron out problems (adding folks to your white list, basically) without throwing out a lot of confirmation requests in the meantime.

[ Parent ]

Upon further consideration... (3.00 / 1) (#174)
by Kyle on Sat Aug 23, 2003 at 03:41:16 PM EST

Having thought about this some more, I think it's maybe not as simple as I said before.

When my friends email me, they're doing me a favor. I love hearing from them. Of course, they're on my white list, and they should not get a challenge.

When strangers email me, and I read their message, I'm doing them a favor. I don't feel a strong obligation to read whatever's sent my way, so the fact that these people are challenged bothers me very little. Hope that, given the effort put into emailing me in the first place, they're willing to put forth the tiny bit more to get the message through the challenge/response cycle.

[ Parent ]

Oh great Kyle, grant me the boon of your attention (4.00 / 1) (#208)
by mozmozmoz on Tue Aug 26, 2003 at 07:30:51 PM EST

I solicit email from people in a variety of ways. I hand out flyers with email addresses on them, I have "email me" links on my web site, I participate in mailing lists and online forums like this one. For that matter, I send out a resume with an email address on it. I still check an account that was used to post to newsgroups, and that still gets a lot of spam.

Now, all of that leads to me getting rather a lot of email from random punters. With no filtering at all I get about 1% spam. By using a fairly basic filter (stripmime tweaked to ignore .doc and .pdf), and by sending forged bounce messages to spam when I can be bothered, I drop the volume of spam below my detection threshold.

I realise that this is not the case for many of you lucky US people, but for me that works. And I know quite a lot of people who do at least some of the email soliciting that I do. Making those people jump through hoops in order to have me read their email isn't really what I want to do.

So I look for ways to recieve email that don't cost the people who are sending it, but ideally cost the spammers more than they cost me. Which I think is how it should be.

Finally, great Kyle, I apologise for this trespass on your time. I am not worthy!

There's lots of comedy on TV too. Does that make children funnier?
[ Parent ]

I'm glad that works for you! (none / 0) (#209)
by Kyle on Tue Aug 26, 2003 at 09:51:24 PM EST

It took me a while to get the "great Kyle" joke. I finally understood what you were driving at when I looked at the comment to which you were replying. ("Great dense Kyle" jokes may now commence.)

If your methods work for you, that's great. I have to say, if your spam problem isn't driving you nuts, you probably don't want something so strong as TMDA to keep it at bay.

I wouldn't want to use TMDA on an account where I was seriously soliciting email (e.g., an address I use for a job search). It's possible, BTW, to use it somewhat selectively. Like SpamAssassin, you can specify incoming addresses that are not filtered (aside from the tagged addresses mentioned earlier). I have a few of those. It's nicer in that they look nicer and can be relayed easily to someone without cut-and-paste. The down side is that I have to maintain the list of them on my side.

[ Parent ]

No (4.25 / 4) (#79)
by awgsilyari on Thu Aug 21, 2003 at 08:26:32 PM EST

You don't bother answering the door when someone knocks either.

No, more like, you knock on someone's door, and this big redneck with a shotgun opens it and yells "Who the fuck are you?"

Same shit with software that makes me jump through hoops to run it, in the name of "preventing piracy." Sure, maybe these guys get ripped off, but they're still treating me like a criminal by default.

I suppose you're the type who would stop giving loose change to homeless people just because most of them are rude and obnoxious. Every so often there's that one homeless guy that just seems nice, talkative, and hey, intelligent. But you never find that out, because you're too busy telling him to fuck off.

Same shit here. Yes, I know you get spam, yes, I have sympathy for that. Its still no excuse to be a xenophobic asshole. It seems nobody is willing to give up a little convenience in the name of hospitality.

--------
Please direct SPAM to john@neuralnw.com
[ Parent ]

No (3.50 / 2) (#84)
by mge on Thu Aug 21, 2003 at 09:46:10 PM EST

It's more like a camera that looks at the person nocking. If they have ever stopped by before, it lets them in. If they haven't, it asks them to nock again, and then lets them in. No shotgus, no readnecks.

[ Parent ]
Gated communities. (3.50 / 2) (#85)
by Kyle on Thu Aug 21, 2003 at 09:49:23 PM EST

Big redneck with a shotgun? I wish I had that for spammers!

I think of a challenge/response system more like a gated community. When you get there, a professional guard asks you to show that you should enter. Unlike real life gated communities, the guard then gives you a pass so that you can walk through the gate unhindered next time.

Gated communities have their place (in a high crime neighborhood, for instance). If a user is getting a tremendous mass of spam, so much that even the .1% that makes it through a good filter is annoying, TMDA is a good solution. If you never get more than a spam per month with the solution you have now, I can understand not wanting to bother people with challenge/response.

It seems nobody is willing to give up a little convenience in the name of hospitality.

You're not willing to give up a little convenience (emailing strangers unhindered) in the name of, what?

[ Parent ]

It's my Door Man (5.00 / 2) (#118)
by ssentinel on Fri Aug 22, 2003 at 12:05:20 PM EST

Try this analogy,

Your linving a block away from the largest vacume cleaner salesman convention on earth. It's been going on for the last 10 years, and they have a daily contest to see who can sell the most vacume cleaners. As a result every day you get 500 or more vacume cleaner salesmen knocking on your door to sell you a vacume cleaner. By now you've got 10 of them, they all suck ;) and you're getting pretty fed up with all these salesmen. Eventually just for your own sanity, you hire a butler to answer the door. Now your butler cordially greets everyone who comes knocking, and only allows people with appointments (those on your whitelist) and those without appointments that have valid business (those who answer the challenge) in. The butler is not rude but he is serving a purpose.

If you don't like the butler scenario replace the butler with a receptionist. The receptionist knows all your appointments, knows what sort of business you conduct, and if someone with an appointment or with valid business calls/arrives, the receptionist sends/puts them through to you.

[ Parent ]

lmao (none / 0) (#164)
by Josh A on Sat Aug 23, 2003 at 07:34:44 AM EST

It seems nobody is willing to give up a little convenience in the name of hospitality.

Need to look in the mirror much?

---
Thank God for Canada, if only because they annoy the Republicans so much. – Blarney


[ Parent ]
BS (3.50 / 2) (#101)
by Eivind on Fri Aug 22, 2003 at 04:33:47 AM EST

It's not that simple. It's not like I have a reasonable choise. My adress(es) exist on MANY websites, they have to, because I *WANT* to be easily contactable, to me that is part of the POINT of email.

This means I get a flood of spam. Typically 500 a day. There's no way I'm going to bother deleting them all manually.

So my solution is to make myself *sligthly* harder for strangers to contact. If they want to reach me, they have to be willing to press "reply" and "send" in their email-program. If not, the message they're sending (I mostly get support-questions for software I'm maintaining from strangers) cannot be that important.

People who have communicated with me in the past, or who is responding to something I sent them, or who is responding, within 14 days, to something I posted on usenet never see any of this.

Yes, ideally I'd not need to force people to jump hoops. However realistically, for me it's not TDMA or free email. It's TDMA or no publically available email adress at all. I prefer TDMA.

[ Parent ]

Excellent. (4.00 / 3) (#99)
by Entendre Entendre on Fri Aug 22, 2003 at 02:43:04 AM EST

Funny thing is, you probably see that as a bug.

Whitelist users, on the other hand, see that as a feature: If your message isn't important enough to hit "reply" when you get a challenge, then your message isn't important enough to read.

--
Reduce firearm violence: aim carefully.
[ Parent ]

Interesting. (2.00 / 2) (#72)
by Vesperto on Thu Aug 21, 2003 at 06:35:27 PM EST

It's good practice, i might try it. For now i just delete the spam and using different email adresses for different purposes. I also try and "goof up" my email adress whenever making it public. Like "myname2000@example.com (please remove the digits)". +1SP

If you disagree post, don't moderate. Alimaniere forf
You forgot about one thing (4.80 / 5) (#74)
by jbuck on Thu Aug 21, 2003 at 06:54:17 PM EST

You forgot about the SoBigF virus (or worm). Because my email address is out there (I used to maintain the GNU C++ FAQ and help with GCC development), I get lots of spam, but because of two layers of very good spam filtering, about 99% is stopped with essentially no false positives: until SoBigF went crazy. I'm now getting about 100 crap mails per day from autoresponse daemons thinking I sent a virus (clue: SoBigF greps for addresses from many sources, sending to random addresses and forging random return addresses).

So, if you install your challenge-response system, it will shower a lot of innocent folks with mail.

I forgot? (3.00 / 4) (#83)
by Kyle on Thu Aug 21, 2003 at 09:42:04 PM EST

Did you read the section called "Spammers forge the address of an innocent victim when they spam, so that bystander will get a huge pile of challenges."? Briefly, this problem is not specific to challenge/response systems, and TMDA's responses are made to be recognized as such.

I get about 250 bogus automatic replies per day because of spammers forging my domain in their headers when they send to addresses that don't work. I filter these out; I never see them. I just looked in the pile of those for today and noticed that someone was sending a .PIF attachment.

Are the 100 autoresponses you got from challenge/response systems, or are they from helpfully configured virus scanners?

[ Parent ]

Very weak counter arguments (5.00 / 2) (#189)
by Eloquence on Sun Aug 24, 2003 at 09:20:08 PM EST

The simple fact is that with TMDA type services deployed on a large scale, you will generate more unnecessary email that will be sent to a very large number of completely unconcerned people, especially because of email worms that forge addresses. It doesn't matter at all that this pile of unnecessary mail is added to the existing pile of unnecessary mail, and it doesn't matter how easy to filter it is (most people you will hit with this are clueless users who do not filter their mail).

This is the killer argument against TMDA. If you want to get rid of spam, use SpamAssassin in combo with Bayesian filtering, scorefiles, whitelists etc. Don't contribute to increasing the noise level on email.
--
Copyright law is bad: infoAnarchy Pleasure is good: Origins of Violence
spread the word!
[ Parent ]

me too! (4.00 / 1) (#201)
by klash on Mon Aug 25, 2003 at 06:20:49 PM EST

I'm now getting about 100 crap mails per day from autoresponse daemons thinking I sent a virus

I'm getting this exact same thing! I haven't decided yet how to cope with it. Any suggestions? I'm considering filtering anything from MAILER-DAEMON or postmaster to a spam folder, as well as anything with "virus" in the subject. But I'm wondering if this is a sound strategy.

[ Parent ]

+1FP, Too /.-centric (3.33 / 3) (#78)
by Random Liegh on Thu Aug 21, 2003 at 08:13:57 PM EST


--
Fives for the funny, one's for the spelling flames, and 0's for the assholes ^W geeks.
Go stuff yourself. (4.33 / 6) (#80)
by it certainly is on Thu Aug 21, 2003 at 08:43:45 PM EST

Over the past year, I've had over 100 people email me directly asking me questions (or reporting bugs, stuff like that), and I've had my reply bounced for whatever reason.

I used to go through the hard work of trying again from different From: addresses to get my hard-crafted replies to their mail through, emailing admins and such. Now I have given up. It's too much hard work.

My strategy now is this: you get a short initial reply from me, roughly answering your questions, explaining you can reply to ask me for more detail if you need it. If my response to you bounces, for any reason other than "mailbox full", you go in my shitlist. I make no further attempt to reply to you. No pleading, no changing my address, no filling out forms or mailing your admin nazi, nothing. You asked me a question, so I'll talk with you, not your fucking doorstaff.

It would be really good if all anti-spam "solutions" made immediately notified you if email destinations are currently blockaded before you send email to them. It would immediately alert you to your problem, and would save me all time and effort I waste into creating replies that will never get read because of overzealous anti-spam "solutions".

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.

So, you like TMDA? (3.66 / 3) (#82)
by Kyle on Thu Aug 21, 2003 at 09:34:37 PM EST

The problems you describe happen with other challenge/response systems (or sometimes with filtering systems, if they take a disliking to you), but TMDA was made with your problems in mind.

If you ever get blocked emailing someone you've never emailed before, it'll "notify" you (with a challenge). With it configured correctly, it should never challenge you if you're replying to someone who's using it.

Your emails don't get silently dropped off the face of the network. You don't get emails from someone who will block you when you try to write back. These are some of the things I really love about the system, and for the reasons you describe.

[ Parent ]

Didn't sound like he liked it. (4.00 / 1) (#142)
by peeping_Thomist on Fri Aug 22, 2003 at 06:57:27 PM EST

but TMDA was made with your problems in mind.

If mail from you to me at one of my addresses is forwarded to me at another address, and I reply to you using the second address, TMDA will bounce my response.

You could counter that it's my responsibility to respond to you from the same address to which you wrote, but when it's a matter of you asking me for help, I'd say it's incumbent upon you to do the work to make sure that our communication succeeds. And that means you can't use TMDA if you want to ask for help from someone who uses multiple email addresses and may not respond from the same address to which you send mail.

In point of fact, I always reply using the address to which mail was sent, but not everyone does.

[ Parent ]

Replies really do work. (3.00 / 1) (#149)
by Kyle on Fri Aug 22, 2003 at 09:38:42 PM EST

Someone else brought up the same question. I answered here, and, to a lesser extent, here. Of course, the best answer is in the FAQ.

I hope this clears things up for you.

[ Parent ]

Nope, (none / 0) (#167)
by it certainly is on Sat Aug 23, 2003 at 08:23:47 AM EST

into the land of zealotry we go.

If TMDA sends any "challenge" messages at all, it has completely failed. It is no good and must be removed. It's a kick in the fucking teeth to anyone who receives one. How would you like it if every single website you visited, no matter what page it was, just to view a page (not sign up for an account, but to provide the basic transaction of HTTP -- delivery of pages, like delivery of email is the basic transaction of SMTP), asked you to fill in a challenge/response to prove you weren't an address harvesting robot?

Even if you don't believe that "every" person would use TMDA, you should at least see what happens to websites that demand hoop-jumping, like the New York Times -- people are openly willing to post to message boards asking for copyright-breaking reposts, direct links and login details rather than jump through any hoops just to see the article. No sane human likes jumping through hoops, and should DETEST jumping through hoops to benefit someone else rather than themselves.

You might think "I'm worth it", and people who want to mail you should jump through a hoop. Well, you're not. I am worth it, however. If you want any technical support for my software, you will NOT use TMDA or any other such system. If I discover you have used it (by getting a challenge), you go on my shitlist and never, ever receive my support again.

I love your "if configured correctly" dreaming. Here's a fucking good idea: if all mail servers in the world were "configured correctly", none of them would be open relays! The amount of spam in the world would drop dramatically, as spammers would have to use their own mail servers to mail out every single spam, as opposed to raping thousands of broken, badly configured open relays. Here's another idea. If nobody replied to SPAM (remember, we're in a fantasy world where pink unicorns roam the land and all anti-spam systems are configured correctly and I never get bounce or challenge messages for replying to people), spammers would give up! Hey! This fantasy, make-believe world is pretty good!

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.
[ Parent ]

...in the wonderful world of Oz. (none / 0) (#169)
by Kyle on Sat Aug 23, 2003 at 09:38:35 AM EST

The New York Times is an interesting case you point out. Some things come to my mind immediately.

  • It's their site and they can do what they want.
  • Copyright violations do not convince them otherwise.
  • If you had an access control on your site, and folks were actively trying to bypass it, would you say, "access control must be a bad idea"? Or would you consider it trespassing?

You correctly point out that many mail servers are badly configured and that this contributes to the spam problem. Do you think that mail servers should not be used because some of them are badly configured?

[ Parent ]

Basically, (none / 0) (#171)
by it certainly is on Sat Aug 23, 2003 at 12:27:27 PM EST

because of the New York Times' website demographics form, I will never read the New York Times' website.

I will read any other news site's take on the same story, including all the NYT's competitors, provided they do not have these stupid, invasive demographics forms. I will not read the NYT's version of a story. If they ever want me to read their stories or look at their advertising, they will have to get rid of their demographics form. That is the bottom line. The sole reason they operate that website is because THEY want ME, with MY eyeballs, to look at THEIR advertising. I will NOT do that until they remove the demographics form, no matter how good the "bait" (stories) are.

I openly encourage people to violate the NYT's copyright, especially if they precede their unauthorised reprint with a phrase like "Here's the NY Times article. Don't read it at their website until they get rid of the unnecessary, invasive demographics form". Because of their demographics form, I have no interest in reading the NYT's stories. However, many people are weak, so reprinting the story for them saves them from having to fill in that invasive demographics form. The more people who are brazenly flaunting it, creating the maximum possible negative publicity against their demographics form, the better. The more people who realise they don't have to put up with that shit, the better. People who already read the NYT website should stop doing so. People who don't read it will understand they simply have to hold out until the NYT abandons their demographics form.

The NYT are not providing their website for the good of their hearts. They provide it to sell advertising. Their "product" is people like me. Their "customer" is marketeers. If they piss off their "product", they will find themselves with no "product" to sell. Any discussion of "it's their site" is doomed to failure. The NY Times doesn't run the site for its own benefit. It runs the site to ensure a continuous stream of eyeballs looking over advertisments.

As for mail servers, if they are badly configured, they should be taken off the internet immediately, and only allowed back on when they stop spewing unwanted spam like "Dave Spart's personal demand that you prove you're not a spammer" emails.

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.
[ Parent ]

How Ironic (none / 0) (#177)
by Ranger Rick on Sat Aug 23, 2003 at 04:19:05 PM EST

That zealotry goes both ways.  In the same token, the only way I would send a challenge e-mail to you is if I've never talked to you before, or you scraped my e-mail off my web site and you're not in my whitelist.

Otherwise, if I sent you an e-mail out of the blue asking a question, the from address would be encoded such that your response would work regardless of what account you get switched to.

When properly implemented, the only time a challenge happens is when I have no previous interaction with you, and you e-mail me.  In that case, it's your responsibility to make it "worth it" as you say.

There's no winning this argument because both sides say "I'm more worth it than you are."

:wq!


[ Parent ]
a thought or two on whitelist clusters (3.71 / 7) (#81)
by martingale on Thu Aug 21, 2003 at 08:55:15 PM EST

Let's consider all people on earth for a moment, (well only all people who have email). Pick two people, and link them together if they are live human beings and they've talked to each other once (via email). You can pick a person, and follow all the links to other people he or she talked to once, and so on. We'll call this a cluster.

There's a lot of clusters, but at most as many as there are people on earth with email. In a cluster, two people may have never talked to each other, but have each talked to a common third person, say. As time goes by, the clusters grow, and their number shrinks.

People tend to be civil in conversation to each other, but one of them is called Wowbagger, and he likes to call people names. At first, he's nice and talks to a few people, and now he belongs to a small cluster.

Unbeknownst to all, Wowbagger can type really fast, and one day decides to insult everyone in his cluster. Obviously, he can only insult those who he's talked to before, and they'll get cross and erase the link to him. Now Wowbagger has a friend who wears black boots, and this friend knows Javascript. He writes a small program which bootstraps a bigger program and so on, as black booted friends do, so that Wowbagger can send innocent looking emails with a small payload which happens to be hidden spyware.

This spyware waits until midnight and sifts through a person's emails looking to discover direct links this person has to other people in the cluster. One day, it sends insults to each linked person by pretending that person sent them. Among these insults, and perhaps through previous "clean" emails, it sent the hidden payload so it can do the same on that person's computer also.

How quickly is the cluster going to be filled with insults? That depends on the diameter of it, ie the mean number of hops to connect any two people in the cluster. Some people think this number is at most 6 for all clusters on average. I'm not so sure, but I'm sure it's not very big. Of course, if we say it's 6, then if we start from both ends, we only need 3 hops on each side simultaneously.

Writing spyware and artificial black booted software is hard work, and the number of those who are prepared to do it is only a few thousand in the world. The really talented black booted ones are less than a hundred. But Wowbagger is obsessed, and he'll pay big bucks for a working system, even if it needs updating once in a while.

If a cluster fills with insults, what are people going to do? Some will break all their links and laboriously rebuild them over time. This requires consent from both ends of the link, lest someone gets upset. Others won't bother, or just can't for other reasons.

It seems that the clusters are only as strong as the weakest person in it. That's not quite true, since only those directly linked to him or her can be insulted. What's really needed is a few well connected weaklings within the cluster, so that most paths connecting two random people can be mapped through these weaklings. I don't know what the optimal number of weaklings is, but it depends on the average distribution of links for each person. These things are often recognized as power laws. A handful of people (the "promiscuous" ones) talk with everybody. It's likely that these people don't get too ruffled over a couple of insults once in a while. They have a high profile, and probably weak computer ski11z. They might even let the telephone repair man quickly check their email on their machine, because it's a nice thing to do.

There's probably a point somewhere in there.

Whitelistees (none / 0) (#104)
by freestylefiend on Fri Aug 22, 2003 at 04:59:47 AM EST

If I whitelist someone, then I am interested in any insults that they send me and probably able to spot out of character behaviour.

[ Parent ]
either (none / 0) (#145)
by martingale on Fri Aug 22, 2003 at 07:39:19 PM EST

1) you don't correspond with many people (I have that problem ;-)

2) you have a lot of time on your hands (I wish I had that problem ;-)

You're basically saying you don't care about sifting through your incoming email to remove any spam and viruses that are sent by people you know. Fair enough, you don't really need antispam products.

[ Parent ]

Well-connected weaklings (none / 0) (#128)
by phliar on Fri Aug 22, 2003 at 12:49:06 PM EST

Right now I use procmail based filtering, with a whitelist (so my friends can talk to me about their penis sizes or mortgages). I know a couple of people on my whitelist to be such "weaklings" -- they even happen to be the "well-connected non-computer-savvy" people like artists and musicians. I do add some friends' friends to my whitelist. Since my whitelist cluster only grows manually, I can control which linked people I will add to my whitelist. (I also don't allow any kind of "active" email; nothing is done automatically except for tossing messages suspected to be spam.) Although I don't currently use TMDA, I might use it for messages already flagged by spambouncer as spam.

(I think there may be a refutation of your possible point somewhere in there!)

Faster, faster, until the thrill of...
[ Parent ]

I think you may be answering the wrong question (none / 0) (#144)
by martingale on Fri Aug 22, 2003 at 07:35:37 PM EST

I know a couple of people on my whitelist to be such "weaklings" [...] Since my whitelist cluster only grows manually, I can control which linked people I will add to my whitelist.
Which means that you are vulnerable to messages (such as viruses and spam) sent by those people ('s computers). Also, the others who regularly correspond with these well connected people are vulnerable too.

I'm not sure I see what you are refuting. Are you saying that you never receive email viruses? Most spam doesn't yet use virus technology, but that's not to say this will never happen. Quite on the contrary (quick google random link), it's the only way to go.

What we are likely to see in the future is all spam appearing "out of nowhere" into the SMTP bloodstream. The spam is sent by trusted people who don't know it. When they become aware of it and fix the problem, some other victim will send some more. Nobody can stop it, and all the identifying information will be either fake or useless (ie the trusted user's credentials, hijacked by the spammer temporarily).

Given this prospect, the only way to filter will be by looking at the whole message (ie content-based, Bayesian or human secretary or otherwise) for each message, and decide on a case by case basis.

Now the whitelist idea says: validate a sender once and trust all email from that sender in the future. It's like having a password protected computer which asks for the login password once at boot time and is left running wide open for a year or two (without crashing ;-), so that anyone can come and use the computer without ever being asked for the password again during that time.

You say you use spambouncer. That's not a whitelist, and isn't vulnerable to the same problem, because each message is looked at on a case by case basis and compared with the criteria. All content-based filters (whatever their quality and other problems they may have) bypass the whitelist vulnerability, precisely if and when each message is scanned and a decision is taken based solely on the message contents, not some stable external information.

[ Parent ]

Bottom line: TMDA works. (3.40 / 5) (#89)
by tmoertel on Thu Aug 21, 2003 at 10:07:32 PM EST

I have been running TMDA since December 2001. It works.

It has identified virtually zero false positives and it kills virtually all spam I receive. Last month I received 3062 spam emails. Courtesy of TMDA, I saw perhaps a handful of them, and they were dispatched with ease. In short, TMDA gave me my inbox back.

With an average of 100 spam emails aimed at my inbox every single day, deleting by hand is not a practical option for me -- I would delete too many legitimate emails. Nor (yet) has Bayesian filtering proved as effective as TMDA, which stops virtually all of the bad stuff while letting virtually all of the good stuff through.

Regarding the "burden" of replying to the challenges, in the 20 months I have been using TMDA, I have encountered exactly one person who felt that replying to the challenge email was too burdensome to follow through. I added that person to my whitelist by hand (it's easy) and it hasn't been an issue since.

As I wondered when I wrote a diary entry in which I analyzed the amount of spam I receive, the arrival rate appears to grow exponentially, doubling every 7.5 months:

    Spams received by month

    Dec 2001 :  410 =====
    Jan 2002 :  480 ======
    Feb 2002 :  494 =======
    Mar 2002 :  845 ===========
    Apr 2002 :  566 ========
    May 2002 :  697 =========
    Jun 2002 :  539 =======
    Jul 2002 :  590 ========
    Aug 2002 :  715 ==========
    Sep 2002 : 1102 ===============
    Oct 2002 :  913 ============
    Nov 2002 :  903 ============
    Dec 2002 : 1099 ===============
    Jan 2003 : 1147 ===============
    Feb 2003 : 1194 ================
    Mar 2003 : 1462 ===================
    Apr 2003 : 1983 ==========================
    May 2003 : 2311 ===============================
    Jun 2003 : 2087 ============================
    Jul 2003 : 3062 =========================================

   
Each = represents 75 emails.

I am sorry that some people feel that challenge-response whitelists like TMDA are inherently bad, but with the spam rate growing as it is, I cannot foresee using anything less effective. I'll keep investigating the Bayesian filters, however. If they become more effective on my particular mix of inbox contents, I'll certainly make the switch. But until then, I'll stick with TMDA.

--
My blog | LectroTest

[ Disagree? Reply. ]


TMDA is the *reason* your spam grows exponentially (4.83 / 6) (#97)
by QuantumG on Fri Aug 22, 2003 at 01:23:27 AM EST

Spammers see any reply from an email as a "hit" and will send you more spam. I hope you see that sending challenges is in-fact encouraging them to send you more spam. This is why we need blacklists as well as whitelists, so that we can stop the spam closer to its source.

Gun fire is the sound of freedom.
[ Parent ]
For Many, The Problem Is Having To Read Spam (4.00 / 1) (#103)
by freestylefiend on Fri Aug 22, 2003 at 04:54:59 AM EST

It sounds as if TMDA addresses this well. It won't prevent me spending resources other than my own effort on spam, but that doesn't matter much if spam doesn't use up my bandwidth, etc.

I don't believe that blacklisting is the only solution to this part of the spam problem. For example, Internet Mail 2000 is an alternative solution.

[ Parent ]

Not (3.00 / 1) (#114)
by Kyle on Fri Aug 22, 2003 at 10:56:30 AM EST

My spam also increases exponentially. It's been doing that since long before I started using TMDA. The observation that it was behaving that way is part of what lead me to look into TMDA in the first place. Before I used TMDA, spammers never got a reply from me.

[ Parent ]

Really? That claim doesn't withstand scrutiny. (4.00 / 1) (#126)
by tmoertel on Fri Aug 22, 2003 at 12:39:13 PM EST

The facts don't agree with your explanation. Your claim is that spammers see a reply as a "hit" and will therefore send me more spam, hence the observed exponential growth rate. However, this claim doesn't withstand scrutiny:
  • TMDA sends its challenges not to the email's From address (the "reply" address) but to the envelope sender of the incoming email (the "send bounces here" address), which for spams is almost always bogus because spammers don't want to receive the massive load of bounces caused by all the bad addresses on their lists. Therefore, spammers almost never see the TMDA challenges and don't know to add me to their high-volume lists.
  • Assuming that the above were not true (but it is) and that spammers routinely moved me to their high-volume lists, this state of affairs would explain only a constant factor of growth, the constant being the ratio of emails sent on the high-volume lists to the low-volume lists.
How, then, do you explain the observed exponential growth rate by my use of TMDA?

--
My blog | LectroTest

[ Disagree? Reply. ]


[ Parent ]
Exponential growth (3.00 / 1) (#157)
by cbraga on Sat Aug 23, 2003 at 01:33:22 AM EST

How, then, do you explain the observed exponential growth rate by my use of TMDA?

The exponential growth is due to your email getting better known. You see, spammers also buy and sell email lists routinely. In fact, lots of the spam I receive advertises lists of emails and spam software. So, if we assume an exponential spread of your email address which is consistent with the speed of spreading being dependent on how many have it, the exponential growth in spam follows.

Also, having a, email client which doesn't load remote images helps a bunch since they'll think your  email isn't valid and give up.

ESC[78;89;13p ESC[110;121;13p
[ Parent ]

No, that can't be the explanation. (none / 0) (#180)
by tmoertel on Sun Aug 24, 2003 at 12:15:31 PM EST

The exponential growth is due to your email getting better known. You see, spammers also buy and sell email lists routinely.
There are two problems with this explanation. First, I have had (and have publicly used) the same email address since 1996. I doubt that there are many address lists it isn't on yet.

Second, this explanation does nothing to explain why the growth rate is exponential as a result of my using TMDA. It is unreasonable to assume that spammers would buy and sell only lists containing 100%, genuine, certified TMDA addresses (or some other class of addresses that contains TMDA addresses and not addresses, say, farmed from web sites). Therefore, if your explanation holds for TMDA use, it also holds for non-TMDA use, and (here's the point) thus supports my original claim that the growth rate of spam in my inbox in general is exponential.

In short, the exponential growth rate is not caused by my using TMDA. (Or, if it is, nobody has offered credible support for such a claim.)

Also, having a, email client which doesn't load remote images helps a bunch since they'll think your email isn't valid and give up.
Oh, I do indeed disallow the loading of remote images from my email clients.

--
My blog | LectroTest

[ Disagree? Reply. ]


[ Parent ]
Boots and suspenders (3.00 / 1) (#96)
by Will Sargent on Fri Aug 22, 2003 at 12:52:54 AM EST

I dunno.  I'd rather accept some level of inconvenience and help a stranger than go through some fascist authentication scheme.

I have a two level scheme.  SpamAssassin on the server takes care of the vast majority of spam and moves it into a folder.  Thunderbird on the client takes care of the remainder.  I eyeball the headers  in the Junk folder before deleting them just to make sure I don't have any false positives.  

Sure, it's more inconvenience, but I get the satisfaction of deleting the spam personally.  YMMV.
----
I'm pickle. I'm stealing your pregnant.

Think. (none / 0) (#98)
by Entendre Entendre on Fri Aug 22, 2003 at 02:40:15 AM EST

I'd rather accept some level of inconvenience and help a stranger...

Like, for example, the inconvenience of hitting 'reply' when you get a challenge, so that a stranger can enjoy not only a spam-free inbox, but also a message from your highness?

Excellent.

--
Reduce firearm violence: aim carefully.
[ Parent ]

Sure. (none / 0) (#117)
by Will Sargent on Fri Aug 22, 2003 at 11:25:31 AM EST

I have no problem with receiving challenges.  

I don't like sending automated e-mails.   And I don't like using a scheme which requires sending them to others.
----
I'm pickle. I'm stealing your pregnant.
[ Parent ]

It requires too much interaction (2.66 / 3) (#100)
by voblia on Fri Aug 22, 2003 at 03:06:19 AM EST

To stop spam we must stop those few who are buying the stuff being offered. I don't think that anyone who *reads* spam is going to use some system that requires active user interference while filtering spam. Yes you won't see the spam being sent to you, but the mail servers will be loaded with spam so the problem is far from solved ...

The perfect filter must be buil into the mail app and require no user interaction or it should be serverside maybe then everyone will stop buying "enlarge your testicles" pills.

Ignas Mikalajunas

Causes traffic and is an annoyance to postmasters (4.80 / 5) (#102)
by wastl on Fri Aug 22, 2003 at 04:49:25 AM EST

I thought the idea to challenge those who want to send me email was good at first. However, there are some serious problems with it.
  • Annoyance to innocent postmasters. Since some time, one of the domains hosted by me is used by some spammer with random user names as forged from address. My system already gets around 20 bounce messages a minute due to non-existant receipients, and the like. If a significant number of users would use this challenging system, this traffic would increase even more. And this IS different than vacation messages, since those are (usually) only sent once to a sender within some weeks and not too many people use them.
  • If a sufficient number of people use this, a large number of spam messages would create yet another message to be delivered over the net, which could greatly increase the traffic caused by Spam
  • There is a large number of people out there who would not know what to do with the challenge message and simply say "this guy's email doesn't work". I worked once in the adminstration of our department at university and, believe me, many people are like that. You say, that is collateral damage. But if you need to do business with those people, then it is YOUR damage.
  • You also say that there can be no "challenge deadlock". However, many people (like me) receive email on multiple addresses and reply only with one. In such a case, it would very well be possible that two people using this system would only be possible to communicate with great effort (changing from address).
Sebastian

Good ponts. (3.50 / 2) (#116)
by Kyle on Fri Aug 22, 2003 at 11:23:53 AM EST

You're right about annoyance to postmasters, but you also point out that they're already annoyed. I get the same bounce messages you talk about, and they have nothing to do with vacation messages or TMDA. A user getting spammed heavily may decide to use TMDA. Postmasters will get the challenges sent to all but the user's friends. The user may decide instead to abandon the account. In that case, the postmaster gets bounces from the spammers and the user's old friends. Is closing an account less acceptible than using TMDA?

It does cause more email traffic. I look at it as the price to pay to get rid of spam. If you see differently, I respect that.

There are indeed folks who won't understand the challenges. I think these same users are sometimes confused when their email is silently dropped by other spam filters. With TMDA, at least they have a clue to go on about why it failed. In short, I like the fact that TMDA fails loudly.

The dated addresses I mentioned in the article should mostly eliminate any problems with people changing addresses. If I send you a message with Reply-To set to a dated address, you can reply from anywhere. You'd have a problem if you were white listed as one thing, I sent you a mail there (with no tagged address), and you replied from somewhere else. To get white listed, though, either I'd do it (hopefully from looking at an email you sent) or you'd do it by responding to a challenge. Either way, you should only be listed with the (one) address you use to send mail.

Let's say, though, that you want to send mail to me from lots of addresses. If you have too many addresses to white list, I could generate a keyed address you could use to email me from anywhere. Other systems that might get in your way don't have the same solution; either they like your email as-is, or I'd have to be able to white list you.

[ Parent ]

Spam and bounced addresses (2.00 / 1) (#119)
by Ta bu shi da yu on Fri Aug 22, 2003 at 12:13:13 PM EST

Surely the TDMA system would decrease overall spam, thus cancelling out the effect of increased bounced messages.

Just a thought.

Yours humbly,
Ta bù shì dà yú


---
AdTIה"the think tank that didn't".
ה
[ Parent ]

SMTP tarpits (4.50 / 2) (#106)
by zenyatta on Fri Aug 22, 2003 at 05:44:48 AM EST

No anti-spam discussion would be complete without mentioning the TarProxy or other tarpit systems. For those unfamiliar with the concept: a tarpit is an SMTP server that filters incoming messages byte-by-byte as they come in. If it detects spam, it throttles that particular connection down to like 1 byte per second, so the cost of sending spam becomes prohibitively high. No mail gets lost yet spammers get screwed big time. There are two obvious drawbacks, though:
  • it's resource-intensive - you can't run a tarpit on a throwaway 386 box
  • for spam to be truly erradicated, all (or at least a vast majority) of SMTP servers on the Net would have to be tarpits
Still, it's an intriguing concept.

One better: teergrube (5.00 / 1) (#160)
by kmself on Sat Aug 23, 2003 at 04:03:46 AM EST

Unless these are the same thing and I'm missing something...

A teergrube is not resource intensive, and accomplishes much the same goal as your tarpit. The idea isn't to throttle the incoming connection, but simply to hold it open. A held connection is one the spammer can't use to deliver mail.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

small difference (none / 0) (#179)
by F a l c o n on Sun Aug 24, 2003 at 09:23:18 AM EST

Teergruben (LaBrea is another) don't look for the content. The idea behind TarProxy is that it's a real-time spam filter.

If it worked perfectly, it would work like this:
* You send a non-spam message, you feel nothing.
* You send a spam message, the connection gets slower, and slower, and slower, until a 300 baud accoustic modem from 1975 looks blazing fast.

It tries to do that in real-time, judging only the content of your mail. That way, delivering spam messages gets awfully slow. If just a few % of the mail servers use such a system, the spammers can either scan for and avoid them (which would give a HUGE incentive for the others to install this, too) or they can stop spamming because it's getting too expensive.

--
Back in Beta (too many new features added): BattleMaster
[ Parent ]

Thanks, similar - forms of "hash cash" (none / 0) (#193)
by kmself on Sun Aug 24, 2003 at 11:02:00 PM EST

Appreciate the clarification.

Advantage of a pure Teergrube is that it's ass-easy to set up. Any random box with direct port 25 access can serve as one, is MX'd to some nondelivery domain, and sucks up spammers' bandwidth. They're not mailservers in the proper sense.

Advantage of a tarpit is that it actively protects the folks behind the tarpit, which is an active mailserver.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Free email service with a TMDA-like system (5.00 / 3) (#108)
by atreyu42 on Fri Aug 22, 2003 at 07:13:44 AM EST

Bluebottle.



10 Megs, POP, Horde and Free.. (5.00 / 2) (#147)
by msimm on Fri Aug 22, 2003 at 09:06:50 PM EST

I was going to mention that too. I've been using them for a few months now and it is a great service (speed was an issue for a while, but that is fixed). Another thing to mention is the accounts are (?) 10 megs and it includes webmail (base on horde) *and* POP. Apparently the company is OSS friendly (at least fond of Horde..) and the individual user accounts are free (more as advertising for corporate sales which is what their focus will be). Great service.

Legends: Awesome FREE Linux and Windows FPS!
[ Parent ]
One major problem (4.00 / 1) (#150)
by greenrd on Fri Aug 22, 2003 at 10:53:00 PM EST

I just started using it now. And I discovered that it doesn't appear to have any special support for whitelisting mailing lists!

That's a tad embarassing for me, since now everyone who posts to a mailing list I'm on (unless they're already on my small whitelist) is going to get a 'please confirm' message from me. I think.

I'm not sure whether this is a good idea or a bad idea. Spam can come via legitimate mailing lists, too. But shouldn't I be able to at least whitelist maling lists that are already spam-protected?


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

There is.. (none / 0) (#175)
by msimm on Sat Aug 23, 2003 at 03:58:54 PM EST

Go to 'Allowed' and scroll down to the bottom and click 'Add'. You can add either individual email address (john@john.com) or entire domains (john.com). There is also a hand feature to import entire lists:
The Bluebottle Verification System allows for the import of entries into your Allowed Senders list from a comma-separated values (CSV) file. Instructions are available for the generation of this CSV from Netscape and Outlook or Outlook Express.
Its white list is pretty well thought out. You'll probably also want to go under 'Options' > 'Advanced Settings' and make sure 'Do not automatically add recipient addresses to Allowed Senders list when sending mail.' is unchecked.

Legends: Awesome FREE Linux and Windows FPS!
[ Parent ]
try it out (4.00 / 2) (#110)
by hephro on Fri Aug 22, 2003 at 08:42:19 AM EST

MailSnare.net offers webmail/IMAP accounts with TDMA. Works reasonably well, but the GUI integration still leaves a bit to desired.

TMDA vs. TDMA (3.00 / 1) (#111)
by hephro on Fri Aug 22, 2003 at 08:44:23 AM EST

darn.... apparently my thoughts are still too Sprint infested...

[ Parent ]
I use TMDA... (4.00 / 1) (#112)
by Ranger Rick on Fri Aug 22, 2003 at 10:06:44 AM EST

...but combined with other filtering. I worked very hard to make sure that challenging is only used as a last resort, when I've exhausted all other avenues for determining whether the sender is human.

The only spam I see is the (very rare) false negative spam that is sent to a discussion list, because I filter those before I pass things to TMDA for sentry. This *only* happens if bogofilter is wrong (and if you've properly trained it, it rarely is) and if SpamAssassin doesn't catch it with heuristics. When I do get one of these, I just drag it to the "blacklist" IMAP folder, and a cron job automatically picks it up and adds it to bogofilter's spam corpus. Basically, the only people that get challenged by TMDA are those that aren't already confirmed to be related to some other mail I already filter to a different folder.

The biggest advantage to this approach is it's very nearly hands-off. It makes the TMDA tagged outgoing mail optional. You can still use date-based whitelisted mails and such for added surety that people won't get challenged, but challenges are already degenerate case.

In the week I've been running it, I've received over 3,000 mails, and sent 10 challenges total (and this is in the midst of SoBig.F!). 2 or 3 went to people who have never e-mailed me before who posted to my public e-mail address listed on my blog, and the rest were spam.


:wq!


My problem with TDMA (2.00 / 1) (#120)
by Merc on Fri Aug 22, 2003 at 12:17:46 PM EST

Is that the acronym is already used, and means Time Division Multiple Access. Since both terms would end up in the data communications field, it is confusing.



Not a problem! (4.50 / 2) (#122)
by Kyle on Fri Aug 22, 2003 at 12:21:46 PM EST

Many people transpose the letters (as you did). It's Tagged Message Delivery Agent (TMDA).

[ Parent ]

Man, I really should join AD (4.00 / 1) (#136)
by Merc on Fri Aug 22, 2003 at 03:06:57 PM EST

(Dyslecixs Anonysoum)

[ Parent ]
It's rude and it's suicidal (5.00 / 8) (#129)
by cicho5 on Fri Aug 22, 2003 at 12:57:04 PM EST

1. It's rude. I write software as a hobby, freeware and/or open-source. I have several apps, about half of which are under active development. I answer 5-10 support emails daily. When my reply gets bounced by a challenge-response system, I don't bother validating myself, because I do feel offended, irrational as it may be. I've given you the software, I've give you the code, I've written the FAQ which you probably didn't read, then I replied to your email and oh, now I'm also supposed to go through hoops to make sure my reply gets to you? Uh-oh.

2. It's suicidal. I make money as a freelance translator, and I get a fair number of jobs through word of mouth. This means email from a stranger (individual or corporate) is actually welcome. I'm not going to deploy a challenge-response system, because I cannot be sure that a potential client will bother to validate themselves. I used to bounce HTML email unconditionally (using procmail), until I noticed I was bouncing job offers. It's as simle as this.

3. The author of the article, while promoting the benefits of TMDA, lets slip a telling remark, viz. " I'm not too concerned about not getting greetings from people I don't know, so this isn't a problem for me." That's the whole point. *Everyone* is not concerned about not getting *some* type of email; the problem is that the article implies that if everyone were as unconcerned about a this or that email, then TMDA would be the ideal solution. But it's a solution based on a fallacy.

Rude (5.00 / 7) (#131)
by phliar on Fri Aug 22, 2003 at 01:07:06 PM EST

It's rude. I write software as a hobby, freeware and/or open-source. I have several apps, about half of which are under active development. I answer 5-10 support emails daily. When my reply gets bounced by a challenge-response system, I don't bother validating myself, because I do feel offended, irrational as it may be.
The same feelings here; if my reply to someone asking for help (about code I've written) bounces, that's it; I delete the bounce and go on with my life, they will not get an answer from me.

However, the article does say that anyone you send email to should get added to your whitelist automatically. If the person asking for help set up a C-R system correctly, I got added to their whitelist when they sent me the question, which means my reply makes it through without any extra work on my part. Conversely, if I ask someone for help, and they have a C-R system, then I will reply to the challenge because that's a small cost for the help I hope to get.

Faster, faster, until the thrill of...
[ Parent ]

Here's a system where it goes wrong (2.00 / 1) (#203)
by simon farnz on Tue Aug 26, 2003 at 09:53:48 AM EST

I have a personal e-mail address, which I take care to limit only to humans. I also maintain one e-mail address per likely spam harvest source. If you e-mail (say) k5@farnz.org.uk, I will reply from my personal address. If you e-mailed asking for help with free software I've worked on, I will probably not bother responding to any challege.

How do you ensure that I can e-mail you without getting challenged? I'm not going to change my behaviour to suit you, and I'm not going to give you a clue about the format of my personal address. I don't even guarantee that it's in the same domain as my public address.
--
If guns are outlawed, only outlaws have guns
[ Parent ]

I guess the article wasn't clear enough. (3.00 / 1) (#204)
by Kyle on Tue Aug 26, 2003 at 11:05:06 AM EST

I answered this before, and there's a better answer in the FAQ.

I'll make you the same offer I made the other user. If you still don't believe, I invite you to try it. Email me your address, I'll make sure it's not on any of my lists, and I'll send you an email as if you're an unknown contact, and we'll see what happens when you reply.

[ Parent ]

My address is in my user preferences (3.00 / 1) (#205)
by simon farnz on Tue Aug 26, 2003 at 11:33:00 AM EST

I publish my e-mail address (or one of them) in every k5 comment.

I challenge you to ensure that:

  1. I won't get challenged if I reply to you.
  2. If I take my time replying (think vacations), I won't get challenged either.
I will reply to you using my work e-mail program, at a future time set by me. If I'm challenged, you fail, since you have pushed the burden of communication, which you want and I am ambivalent about, onto me.
--
If guns are outlawed, only outlaws have guns
[ Parent ]
Take your time? (3.00 / 1) (#206)
by Kyle on Tue Aug 26, 2003 at 12:28:33 PM EST

So, is that an invitation to email you?

Of course, if you take long enough, the dated address won't work. How long you need to wait to prove yourself right is easy to determine, but I'll tell you anyway: my dated addresses last two months.

I don't know how much vacation you get in a year, but I wouldn't want to hold my breath until I got two months off of work. I also wouldn't place bets on me remembering I'd even emailed you two months down the line.

Come to think of it, there's no reason not to make them last a year. I've never gotten spam through one of them. Will you still remember to make your point in a year?

[ Parent ]

It is (none / 0) (#210)
by simon farnz on Wed Aug 27, 2003 at 05:32:21 AM EST

I'm curious to see how well the system works from an end-users point of view. The only time I've come across C-R systems in the wild has been when the muppet user has either not used dated addresses, or set them to an unreasonably short timeout (12 hours was not atypical). Further, the challenge message that came back was rather offensive. Phrases like, "I'm fed up of dick-head spammers. Don't waste your fucking time unless you're human." do not convince me to respond to a challenge.

This is an opportunity to convince me that TMDA is not as bad as the others I've encountered; I still feel that C-R is not a good solution, but I'd like to see if this incarnation actually works from my point of view as a non-user of C-R.

FWIW, my cloud-cuckoo land solution to spam is to make everyone use OpenPGP. I will then accept any e-mail encrypted to my key, and can choose to allow through e-mails signed with certain trusted keys (such as my girlfriend's key). This has certain real-world issues to sort out, but I prefer it to C-R.
--
If guns are outlawed, only outlaws have guns
[ Parent ]

I agree... (none / 0) (#212)
by Ranger Rick on Wed Aug 27, 2003 at 02:13:16 PM EST

The people with those types of challenge messages probably *want* to scare off people who would automatically ignore them, or at least that's my guess.

On the other hand, this is what I put in my auto-response, and I think it pretty much says what my position is:

(This message is auto-generated.)

My e-mail address has been public for a very long time.  Because
of this, I get a *ton* of SPAM (on the order of at least a hundred
SPAM messages a day, if not more), and it is becoming unmanageable
even with SPAM filtering.

If you are seeing this message, this means it is the first time
you have contacted me at this address, and I have never
previously sent correspondence to you.

I understand it is an imposition to have to go through this, but
since this is the first time you have e-mailed me, I ask that you
respond to this mail to confirm you're a real person, and not an
automated SPAM robot.  You will not ever have to do this again --
once you have confirmed, your e-mail will automatically be
accepted to my mailbox.

I also understand if you choose to ignore this, and just not
correspond with me; I realize that I'm handling SPAM at the
expense of people who haven't e-mailed me before, but I've also
done all I can to minimize the number of these e-mails that have
to go out by using a tools to automatically whitelist addresses
to which I've corresponded, or those who have posted to public
discussion lists I frequent.  The alternative is a consistent
sink on my time that could be better spent..

------------------------------------------------------------------
How To Respond
------------------------------------------------------------------

To release your message for delivery, please send an empty message
to the following address, or use your mailer's "Reply" feature.

   [foo]

This confirmation verifies that your message is legitimate and not
junk-mail. You should only have to confirm your address once.

If you do not respond to this confirmation request, your message
will not be delivered.

Your original message is attached below.

:wq!


[ Parent ]
One last thing (none / 0) (#211)
by simon farnz on Wed Aug 27, 2003 at 08:33:45 AM EST

I can think of a circumstance in which I'd contact you from a new e-mail address long after your timed address has expired. This is when I've carried on an e-mail dialogue with you, and suddenly lost access to the address I used to use. Doesn't happen very often, but if it did, I'd be pissed off at being called a spammer, and probably just give up on you.

OK, so it's a rare scenario, but it has happened to me (account closed without appeal when someone did a joe-job on it), and I can rarely be bothered to reply to challenges. Too much pain for the gain I get; it's simpler to just drop you. Granted, you may be happy to lose correspondents in this fashion. Not all of us are, which is why TMDA is an imperfect solution.
--
If guns are outlawed, only outlaws have guns
[ Parent ]

Done the experiment (none / 0) (#216)
by simon farnz on Thu Aug 28, 2003 at 12:21:05 PM EST

I got Kyle to e-mail me, and then replied using the most broken e-mail system I have access to (my employer's system). I didn't get challenged, so it worked in that sense.

In the end, the choice of anti-spam system depends on whether you're more worried about occasionally challenging someone who you shouldn't challenge (e.g. when a friend changes their address), or whether you'd rather deal with the spam manually, and thus guarantee that you never accidentally challenge someone who you'd rather not challenge. I fall into the latter category, and use SpamAssassin and similar products to aid my decision. People who use TMDA fall into the first category.
--
If guns are outlawed, only outlaws have guns
[ Parent ]

I agree (4.00 / 3) (#132)
by blakdogg on Fri Aug 22, 2003 at 01:12:11 PM EST

> 1. It's rude.  <--snip-->
In the case you outline the original sender should've added you to his whitelist. Ideally anyone you send email to should be on your whitelist, at least from my point of view
Woe be onto the United Nations, there nothing but a front.
[ Parent ]
A couple of things. (4.00 / 2) (#133)
by Kyle on Fri Aug 22, 2003 at 01:17:50 PM EST

A properly functioning TMDA installation should never challenge your reply to a message the TMDA user sent you.

If your email is mostly from people who've never emailed you before, a challenge/response system definitely has a bigger down side. I use TMDA on my personal account only. On addresses that I expect to get mostly mail from strangers (e.g., admin role addresses), I don't use TMDA.

You note correctly also that if you're looking for a job, you always want your filtering to be lenient. You'd hate to false positive something so important.

I don't understand your last remarks. What's the fallacy that the solution is based on? The TMDA home page lists four assumptions that drive its design. I've pasted them below. Is one of these what you're talking about?

  1. You cannot keep your email address secret from spammers.
  2. Content-based filters can't distinguish SPAM from legitimate mail with sufficient accuracy.
  3. To maintain economies of scale, bulk-mailing is generally:
    • An impersonal process where the recipient is not distinguished.
    • A one-way communication channel (from spammer to victim).
  4. SPAM will not cease until it becomes prohibitively expensive for spammers to operate.


[ Parent ]
Multiple addresses (4.50 / 2) (#146)
by CtrlBR on Fri Aug 22, 2003 at 07:53:54 PM EST

I have some address in the form me@somefreesoftware.org that is forwarded to my real address.

Guess what?

When I reply I don't have been added to the whitelist, because the software has no way of handling a case like that, that is far from uncommon.

Broken by design.

If no-one thinks you're a freedom fighter than you're probably not a terrorist.
-- Gully Foyle

[ Parent ]
I triple DOG dare you! (3.00 / 1) (#148)
by Kyle on Fri Aug 22, 2003 at 09:32:44 PM EST

I covered this briefly in the article and linked to a FAQ entry with a full explanation.

When I send email to someone not on my white list, Reply-To is set to a dated address which will pass through TMDA unchallenged until an expiration date. By default, that expiration date is five days in the future, but I have mine set to last two months. Anyone can send to that dated address, from anywhere, until it expires. If you try to send to it after it expires, you get a challenge.

(If you're doing this at home, and you think any expiration date is too soon, even one 20 years away, you can send with an address that never expires. Normally, they'd all be the same address, but with some fiddling, you could make them unique.)

If I email you at some alias that's not on my white list, you will be able to reply from anywhere with the dated address. If I email you at some alias that is on my white list, well, I talked about that here.

If you still don't believe, I invite you to try it. Email me your address, I'll make sure it's not on any of my lists, and I'll send you an email as if you're an unknown contact, and we'll see what happens when you reply.

[ Parent ]

Hum. Almost convinced... (4.00 / 1) (#163)
by CtrlBR on Sat Aug 23, 2003 at 06:48:49 AM EST

TMDA seems to do the right thing but either people aren't using it right or there are really sucky alternative that people use out there.

But this system seems to forbid sending an email from your work and reading the answer at home...

If no-one thinks you're a freedom fighter than you're probably not a terrorist.
-- Gully Foyle

[ Parent ]
Well, kinda. (4.00 / 1) (#165)
by Kyle on Sat Aug 23, 2003 at 08:18:01 AM EST

Indeed, there are lousy alternatives.

You want to send from work and read the reply at home? Why is that hard? If I wanted to do that, I'd set the work's email to some tagged address that would work from anywhere, but admittedly, I'd have to do that manually. The easier way is to log into the work network from home and access my work email that way, but I recognize that not everyone has that option.

[ Parent ]

Easy to reduce the number of confirm requests (4.66 / 6) (#138)
by koreth on Fri Aug 22, 2003 at 03:22:42 PM EST

I have my mail set up such that only "maybe spam" messages get sent to the C-R system (a homegrown system sort of like TMDA) and it works beautifully. Very few real first-time correspondents have to reply to a challenge -- I measured it at about 2% a while back -- and I'm able to use more aggressive "maybe spam" rules because the downside of a false positive is so much lower. My "maybe spam" rules include a Bayesian filter, keyword detection, and some other stuff.

"Just delete it" isn't a realistic option for me; I'd never get any work done with my inbox beeping at me constantly (and I get time-sensitive messages at work, so I can't just put off checking my mail.) Here's my monthly spam count for the last couple years:

07/2001 426 **
08/2001 560 ***
09/2001 584 ***
10/2001 749 ****
11/2001 998 *****
00/2002 1405 ******
01/2002 1340 ******
02/2002 1161 *****
03/2002 1345 ******
04/2002 1934 *********
05/2002 1760 ********
06/2002 1797 ********
08/2002 2365 **********
09/2002 2535 ***********
10/2002 2698 ************
11/2002 2881 *************
12/2002 2633 ************
01/2003 2844 ************
02/2003 2392 ***********
03/2003 3406 ***************
04/2003 3946 *****************
05/2003 4604 ********************
06/2003 5054 **********************
07/2003 5702 ************************
08/2003 4399 ******************* (with 10 days remaining)

I try to minimize the impact on legit senders by only auto-challenging messages that might be spam, but really, the occasional lost message from someone whose mail isn't important enough to them to answer a challenge is better to me than to spend an hour and a half a month "just hitting delete, it only takes 5 seconds." That's 95 minutes' worth of 5-second deletes for July, and of course the real cost is more than 5 seconds since it's an interruption to my train of thought.

If that's the case... (3.00 / 1) (#139)
by virtualjay222 on Fri Aug 22, 2003 at 04:18:34 PM EST

Wouldn't that be putting a greater burdon on the spammer's resources, since now they're sending out "exponentially" more e-mails without even the hope of them being read. Thus the ratio of mail sent to products sold becomes awkwardly lopsided.

The only disadvantage I can see is the bandwith that gets chewed up by this whole mess. But personally, I would rather lose bandwith than waste time. Then again, thats only my opinion.

---

I'm not in denial, I'm just selective about the reality I choose to accept.

-Calvin and Hobbes


This is just a work-around for the real problem... (4.00 / 2) (#143)
by curunir on Fri Aug 22, 2003 at 07:22:02 PM EST

An actual solution would be HashCash (integrated into the SMTP protocol.)

Solutions like these simply distract people from dealing with the root of the problem. The only reason I see for using a system like this is if the email address will be listed publicly (e.g. whois record or on a webpage.)

Oh god. (1.00 / 1) (#152)
by valeko on Fri Aug 22, 2003 at 11:08:44 PM EST

I will now proceed to roll my eyes so far back into my head that I will gaze at the stem of my own optic nerve.

Why the hell would you use an 'anti-spam' solution? The very definition of such a thing is that you're wasting more time than you're supposed to be saving. Instead, do as I do; go into pine, and before reading your real mail, take about ten seconds to delete the spam by repeatedly hitting arrow and D keys. Then, re-open pine, and commence with mail reading.

End of story. Why screw with the basic premises of e-mail?

"Hey, what's sanity got going for it anyways?" -- infinitera, on matters of the heart

Open source is only free if... (4.00 / 1) (#153)
by greenrd on Fri Aug 22, 2003 at 11:13:35 PM EST

- sorry - Deleting tons of spam is only free if your time[*] is of no value.

[*] not to mention emotional balance


"Capitalism is the absurd belief that the worst of men, for the worst of reasons, will somehow work for the benefit of us all." -- John Maynard Keynes
[ Parent ]

What exactly is a "ton" of spam? (3.00 / 3) (#154)
by valeko on Fri Aug 22, 2003 at 11:15:41 PM EST

You'd have to get a Himalayan avalanche of e-mail for deleting spam to be an issue, unless you're the victim of some spam-producing virus. It doesn't take long to delete some spam. If I get 70 messages over 3 days, at least 25 of them are bound to be spam, and it's not that hard to hit the D key 25 times.

If you've got problems with mail volume, you should be sifting your mail into folders somehow, anyway.

"Hey, what's sanity got going for it anyways?" -- infinitera, on matters of the heart
[ Parent ]

That's nice... (4.66 / 3) (#158)
by Ranger Rick on Sat Aug 23, 2003 at 01:42:07 AM EST

...but I probably get 70 messages an *hour* if not more.  I'm on a bunch of public developer lists, and I've had the same e-mail address for about 7 years, so I'm on every spam list known to man.  You hit a point where it's difficult to read real mail because all you've got is spam.  In other words, some people *do* have a Himalayan avalanche of e-mail.

Sifting into folders (which I do) just spreads it around among 14 different places, if you're not doing spam filtering.

:wq!


[ Parent ]
Ouch! (3.00 / 1) (#166)
by Kyle on Sat Aug 23, 2003 at 08:21:56 AM EST

So, what's your solution, and how well does it work?

[ Parent ]

My Solution (4.00 / 1) (#176)
by Ranger Rick on Sat Aug 23, 2003 at 04:09:55 PM EST

I mentioned it in another post here, but the summary is on my blog. It's definitely not for everyone, it's fairly involved. But it works great, and sending out TMDA messages is used only as a last defense. Anything else that I can be sure of (both positive an negative) are caught by a combination of bogofilter (bayesian filter), spamassasin, whitelists, and mailing list procmail rules. I have scripts to automatically take messages that I drag to special folders (whitelist, blacklist, etc.) and feed them to the appropriate stop in the rules as needed, so it gets smarter without me having to do anything other than just use my mail client normally.

I used SpamAssassin previously, but it goes through a decent amount of "bit rot" (spammers learn to work around things SA knows about as time goes on) and you end up needing to be on a constant upgrade train of rules and other such things to keep it reasonable. Even then, it ends up letting a decent amount through because I'm not willing to allow for the chance of false positives, so I had set the threshold a bit higher.

Now, even with the volume of mail I receive in a given day, maybe 1 or 2 spam make it through, and so far (knock on wood) I've had no false positives.


:wq!


[ Parent ]
How about 900 messages? (none / 0) (#161)
by kmself on Sat Aug 23, 2003 at 04:12:37 AM EST

That's my count of SOBIG.F virus mails I'v received in the past two days. Friends report thousands.

Fortunately, all of these are classified, with my spam, into a spam folder. Quick scan to see if they are indeed spam. Then one, just one, delete.

Benefits of filtering (SpamAssassin + procmail, if you ask): Spam, and other mail, are classified into folders. I've already got a good idea of what I'm looking at. If it's not in my spam folder, chances are very good it's not spam.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

The avalanche is well underway for some of us (4.00 / 1) (#194)
by koreth on Mon Aug 25, 2003 at 03:20:07 AM EST

Try 300 spams in one day, which is what I get on a busy day right now, and see if it's still so easy to just casually hit delete and not worry about it. Especially when many of them use subject lines like "Hi!" or "Remember me?" that make you unable to identify them using your table-of-contents view of your inbox.

[ Parent ]
Ten seconds? (4.00 / 1) (#155)
by Kyle on Fri Aug 22, 2003 at 11:17:50 PM EST

I got about 60 spams per day last month. Some people get more than that. If you're not having a spam problem, then you don't need an anti-spam solution. If you get spammed moderately, a content-based filter will probably work well enough for you. If you get spammed heavily, TMDA will work wonders. If I did not have some anti-spam software, my email would be useless. If you don't have that problem, I'm happy for you, but that's a far cry from the problem not existing.

[ Parent ]

Hmm. (2.00 / 1) (#156)
by valeko on Fri Aug 22, 2003 at 11:26:50 PM EST

I still don't think I'd go through the trouble, even if I got 100 spam letters per day.

"Hey, what's sanity got going for it anyways?" -- infinitera, on matters of the heart
[ Parent ]

The problem with the delete key (5.00 / 2) (#172)
by vadim on Sat Aug 23, 2003 at 01:50:38 PM EST

is that when you start getting big amounts of spam, like say, the 100 messages a day you suggest, the delete key not only makes you waste time, but it's dangerous too.

A lot of concentration is needed to separate 5 real emails from 100 spams, and chances are, you're going to delete 1 or 2 of the good ones by the time you're done.

Setting up spamassassin is maybe 15 minutes one day. Removing 100 spams every day is about 5-10 minutes a day, depending on how careful you are.

Lately spam even comes with subjects such as "Re: Hi!", which makes it even harder to separate it.
--
<@chani> I *cannot* remember names. but I did memorize 214 digits of pi once.
[ Parent ]

Good filtering MORE accurate than 'delete' (5.00 / 1) (#202)
by kmself on Mon Aug 25, 2003 at 07:47:24 PM EST

The advantage of a good filtering system is that it's more accurate than manually working through mailboxes and folders, deleteing stuff by hand. I know my own accuracy is lower than SpamAssassin. I've seen several comments that "I can't use spam filters because the risk of deleting legitimate mail is too high"....from technology magazine editors (OK, maybe not the best example of clueful users -- and this was a Windows-centric publication).

My response: you're risking more now.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

I adopt a similar approach (4.00 / 1) (#197)
by daragh on Mon Aug 25, 2003 at 08:28:36 AM EST

About one third of my email is spam. So every morning I sit down and randomly delete one third of my mail. I read and react to the rest. Not only do I have less to do in work (my boss seems to get through to me less these days), I now have a larger, harder penis, an improved memory, and have earned upwards of $50,000 in three months from the comfort of my home using only my computer.

No work.
[ Parent ]

Challenge-Response considered harmful (5.00 / 12) (#159)
by kmself on Sat Aug 23, 2003 at 03:51:18 AM EST

This is a repost of my own essay, Challenge-Response Anti-Spam Systems Considered Harmful. Following a really pathetic thread on debian-user, I finally wrote an essay I'd been kicking around for a year or so.

Spam is a growing, heck, exploding problem. No doubt. Challenge-response (C-R) is a flawed tactic, for the following reasons.

0. Weak, and trivially abused, verification basis.

Even where used, C-R systems are readily bypassed by spammers.

The 'FROM:' header of email can be, and routinely is, spoofed. It offers no degree of authentication or evidence of identity.

C-R uses the "From:" header (with implementation-specific variations) as an authentication key. While a given key is going to have a relatively low likelihood of being cleared by a given user, there are keys which will have a high likelihood of being cleared. Off the top of my head, @microsoft.com, @aol.com, @ebay.com, @*.gov, and other major commercial, financial, and governmental institutions, would be likely to be cleared by a large number of users. Similar "social engineering" tactics are already used by spammers.

C-R moves you back to square one of the fact that SMTP can't provide authentication of email headers. At the very least, contextual analysis of headers (as Alan admits) is necessary. If you're already taking this step, heuristic and Bayesian methods are a low-overhead next step, which have proven to be highly effective and accurate.

By contrast, systems which utilize multiple metrics -- sender, header integrity, content, context, Bayesian analysis -- provide a broader, deeper, richer set of metrics on which to gauge spam.

1. Mistaken interpretation of anti-spam goals

The intent of a practical anti-spam system is not to ensure, at all costs, that no spam should darken the reader's inbox at any cost. If that's the goal, then unplugging your computer is the simplest fix.

At a practical level, the goal is to minimize the amount of spam received, while ensuring no (or the very minimum) of legitimate mail is lost. Inconvieniencing spammers is a plus. It is currently possible to achieve rates of a very small handful of spam messages per week via a mix of whitelisting and content-filtering systems, with Bayesian filters attaining very high and accurate rates.

C-R systems in practice achieve an unacceptably high false-positive rate (non-spam treated as spam), and may in fact be highly suseptible to false-negatives (spam treated as non-spam) via spoofing.

2. Misplaced burden.

Effective spam management tools should place the burden either on the spammer, or at the very least, on the person receiving the benefits of the filtering (the mail recipient). Instead, challenge-response puts the burden on, at best, a person not directly benefitting, and quite likely (read on) a completely innocent party. The one party who should be inconvenienced by spam consequences -- the spammer -- isn't affected at all.

3. Privacy violation.

A record of our correspondence is being maintained by a third party who has no business knowing of the transaction. Many people will refuse to respond to C-R requests for this reason.

Virtually all C-R systems must be implemented on the mail server -- putting them effectively _out_ of the immediate reach of the casual home email user, and putting critical information of the email habits of both yourself and your correspondents in the hands of a third party.

Most of the _general_ discussion (that is, outside this mailing list) has concerned service-model enterprise models in which C-R is provided and hosted by a third-party, which is then aquiring a rather interesting database of communications patterns, which _must_ be maintained on a persistent basis. Not the sort of thing I'd like to have available to an arbitrary subpeona request.

4. Less effective at greater burden than reciever-side whitelisting.

A C-R system is essentially an outsourced whitelist system. The difference between a C-R system and a self-maintained whitelist is that the latter is:

  • Maintained by the mail recipient, rather than a third party service provider.
  • Is the responsibility of the mail recipient, rather than the sender.
  • Places the burden on the recipient to add new addresses to allow/deny lists.

I might add that I myself use a mix of whitelisting and spam filtering (via SpamAssassin) to filter my own mail with a very high level of accuracy, in terms of true positives, true negatives, false positives, and false negatives. Namely: better than 98% true positive (filtered spam), less than 2% false negative (unfiltered spam), 99.98% true negative (unfiltered non-spam), and less than 0.02% false positive (filtered non-spam). While some C-R proponents claim filtering doesn't work, it clearly does.

5. High type II error (beta).

Because of numerous issues in sender-compliance with C-R systems, C-R tends to a high false postive rate. This is known as type II error, in statistical tests, and is denoted by beta.

The mechanics of C-R systems lead to a fairly high probability that users of such systems will find themselves missing an unacceptably high rate of non-spam (aka "ham") mail, possibly with very high significance (e.g.: client, commercial prospect, or family communications).

In a staggering display of transrational behavior, C-R proponents frequently and vociferously blame this failure of C-R on the unwillingness of bystanders to be drawn into the misguided system.

C-R systems assume all mail to be spam until proven otherwise. A rational system assumes mail to be of _unknown_ quality, until determined to be spam or non-spam. If mail processing can't determine the mail's quality, it is treated as "grey". Such "greymail" generally amounts to a small handfull of messages daily, even for heavy mail users, and can be readily evaluated, with whitelists and spam filters trivially updated.

For a description of statistical type II errors, see:

6. Potential denial of service.

C-R systems can be used intentionally or otherwise in a denial-of-service or "Joe Job" attack on an innocent third party. In fact, this is likely to start happening shortly as C-R becomes more widespread.

How? Simply: Spammer spoofs a legitimate sending address (this is already commonplace). C-R systems then send out a challenge to this address. With only 1% penetration of C-R, the victim of the C-R/Spam attack is deluged with 100,000 challenge emails. This could likely lead to lawsuits or other legal challenges.

7. C-R - C-R deadlock

This is almost funny.

How do two C-R system users ever start talking to each other?

  • User A sends mail to user B. While user B's address is then known to A, user B's C-R server's mail is not.
  • User B's C-R system sends a challenge to A...
  • ...who intercepts the challenge with A's C-R system, which sends a challenge to user B's C-R system...
  • Rinse, wash, repeat....

No, I didn't think this one up myself, see Ed Felton's "A Challenging Response to Challenge-Response"

Bypassing this deadlock then opens an obvious loophole for spammers to exploit.

While _some_ C-R systems may avoid this particular pitfall, current experience with vacation responders and spam-notification filters provide strong empirical evidence that a significant number of C-R systems will in fact _not_ get this right.

This and several following issues are often countered with "but a well-designed C-R system won't do that". Unfortunately, there will be, and are, many poorly-designed C-R systems.

8. Potential integration into spam email harvest systems.

One commonplace piece of advice for avoiding spam is to not respond to opt-out, aka email validation testing, requests.

C-R spoofing on the part of spammers would simply hijack a presumption that C-R requests were valid to provide spammers with higher-quality mailing lists. See the current rash of identity theft / CC theft scams based on "updating your account information".

C-R at best promotes bad personal identity protection practices.

9. Likely consequences: C-R messages and users blacklisted or spamfiltered

The C-R user is likely to find their own address added to blocklists from many users and/or mailing list adminstrators burned by malformed, or simply unwanted, C-R requests. Simply: people who receive such requests are very likely to just add the sending address, or user corresponding to the request, to their own personal blacklists. This is my own current M.O. with C-R requests, and andecdotal evidence suggests it's a common practice.

This factor is entirely outside the bounds of the C-R system; it is a reflection of the independent response of individuals and organizations to receiving C-R challenges. C-R definitionally cannot accomodate this.

Another possibility is that, due to user concensus, spam filters simply tag C-R messages as spam, either with a direct rule or as a result of Bayesian weighted scoring.

Beyond any semiotic arguments of what spam is or isn't, if the operational reality is that Spamassassin reflects the opinion of SA users and developers and treats C-R transactions as spam, it is for all intents, spam.

10. Mailing list burden.

C-R systems typically misfunction on mailing lists in one of two ways, neither of which is acceptable:

  1. The C-R sends a challenge to the list for messages received.
  2. The C-R sends a challenge to each individual listmember for the first post received.

In both cases, the burden is placed on a party who could care less about the benefits of the C-R system. Several lists of my aquaintance have taken to permanently banning any users who exhibit use of misconfigured C-R systems.

11. Fails to address techno-economic underpinnings of spam.

Spam exists for one reason: it's profitable.

It's profitable because technology allows the costs of sending a large number of mail messages to be lower than the revenues available for doing so.

Any effective spam remedy must attack one or the other side (or both) of this equation: raise the costs or reduce the technological effectiveness, on the one side, or reduce revenues on the other.

C-R, as with most recipient-side filtering systems, imposes negligible incremental overhead on the spammer. A delivery is made, the spam server moves on, the cost is a single SMTP connection for a fractional second. Collateral costs are high: for legitimate senders, spoofed reply addresses, mailing lists, and retaliatory actions on the C-R user.

A truly effective spam defense must attack the techical and economic aspects, in as unobtrusive a manner as possible.

The one system which seems to best fit this requirement is the Teergrub -- the spam tar-baby, FAQ at:

http://www.iks-jena.de/mitarb/lutz/usenet/teergrube.en.html

A teergrubing mailserver costs a spammer multiple SMTP connections, an inherently finite resource, for possibly hours. Workarounds on the part of the spammer are possible, but all result in higher costs, reduced delivery, or both. The net effect is essentially a delivery payment requirement, though the payment is in the form of time and configuration on the part of the spammer. Collateral damage is low -- if a teergrube _does_ unintentionally filter a legitimate sender, the only cost is a single (or very small number of) delayed delivery. This and other issues are covered at the FAQ above, read it before posing hypothetical problems.


--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.

Some thoughts I had; (3.33 / 3) (#162)
by zcat on Sat Aug 23, 2003 at 05:53:31 AM EST

Most of the mail I send is in reply to requests for help. Occasionally I've encountered a C-R system, and I figure what the hell, someone else will have replied. What do I care. Never look a gift horse in the mouth.

However; if the system became common enough I'd probably set up a procmail rule to reply to the challenges automatically. But they all work differently; some send replied from a different address; some send the subject line; some send the message ID. The easiest way to do this is reply to them all blindly, and sites who handle a lot of mail are quite likely to do this. So even if support@microsoft.com isn't already whitelisted; it will be..

[ Parent ]

It's an article in an article. (3.40 / 5) (#168)
by Kyle on Sat Aug 23, 2003 at 09:23:29 AM EST

0. Weak, and trivially abused, verification basis.

I address this partially in my post. I can see lots of people white listing microsoft.com or ebay.com or similar. TMDA provides a solution with its tagged addresses, since challenge/response and a white list are not the only tricks up its sleeve. SpamAssassin (for instance) has a white list also, but if a spammer abuses it (as you say they already do), there's no easy way out. With TMDA, I can go to MS, tell them my address changed to a tagged address, and take them off my white list.

1. Mistaken interpretation of anti-spam goals

I'm confused. Are you telling me I don't have the goals I do?

2. Misplaced burden.

We've discussed this at length already. I think this comment is a decent summary of how I feel about the issue, but I understand reasonable people can disagree.

3. Privacy violation.

I run TMDA on my own mail server, so this isn't a problem for me. I can't help but wonder if you think that ordinary mail logging is also a privacy violation. Should we lobby for our ISPs to turn off the logs generated by their mail systems?

4. Less effective at greater burden than receiver-side whitelisting.

My own experience is that TMDA is more effective than what I was using before. Perhaps filters could have been more effective than they were, but that would have required more fussing with them than I wanted. I fuss with my white list some also, but it's pretty simple. I'm not playing puzzle games with some spammer in an arms race. I also think there's a psychological benefit to fussing to let the good stuff in vs. fussing to keep the bad stuff out. I'm not cursing while I'm fussing.

5. High type II error (beta).

I don't get this. Maybe I need to read the links. What do you mean by "issues in sender-compliance"?

6. Potential denial of service.

I address this in the article and in an earlier comment. Briefly, this problem is easily found in other aspects of the system, and simply closing an account creates a larger problem than using TMDA on one.

7. C-R - C-R deadlock

This is addressed in the article and an earlier comment. Basically, TMDA doesn't have this problem.

This and several following issues are often countered with "but a well-designed C-R system won't do that". Unfortunately, there will be, and are, many poorly-designed C-R systems.

Again, cars are a bad idea because lots of them don't work right. Software is a bad idea because so much of it is poorly designed.

8. Potential integration into spam email harvest systems.

I addressed this in the article. Briefly, doing this is already possible and doing it increases the cost of sending spam. The former is no loss, and latter is a win.

9. Likely consequences: C-R messages and users blacklisted or spamfiltered

As you suggest, perhaps it can't be helped. It looks a lot like other collateral damage, though. I was blocked by a RBL long before I started using TMDA.

In my article, I acknowledge that challenges might get filtered, and this is a total failure of the system. However, I regard it as a bad interaction, not a fault of either system involved. Spam filters sometimes drop the wrong thing. Challenges are sometimes sent that won't be answered. Both systems have ways of dealing with these problems (typically in the form of a spam-filled "suspect" area the user can peruse to find legit mail).

10. Mailing list burden.

TMDA doesn't have that problem.

11. Fails to address techno-economic underpinnings of spam.

To bypass TMDA effectively (with intelligent white list exploitation or a challenge auto-responder) raises the cost of sending spam.

As a last note, I think that tar pits are a pretty good idea (as long as they don't totally halt delivery), and I might run one in the future (along with what I use now). I think it's great how anti-spam solutions that work along different lines can be used together.

[ Parent ]

C-R defenses rebutted (5.00 / 3) (#170)
by kmself on Sat Aug 23, 2003 at 11:07:17 AM EST

First: I'm not restricting my discussion to TMDA, but C-R in general. The proponents of TMDA claim it's the best-of-breed C-R system. Which, accepting your assertion, means all other systems are worse. The claim that "a well designed system won't do that", for a given complaint against C-R, is woefully naive, though charming.

TMDA is a minority of the C-R requests I've received. Both direct and anecdotal evidence in C-R systems (from list operators, the Debian Bug Tracking System, postmasters, etc.) suggest that poorly-behaved systems are the norm. Given evidence of vacation messages (see above) I strongly suspect this to remain the case.

Second: The claim of C-R proponents is that content-based filtering is not effective. In my own and anectdotal experience, this is simply false. Moreso: C-R is effectively narrow content-based filtering reduced to simply the "From:" header of an incoming mail message. Why not availe yourself to the rest of the bloody message, via a comprehensive content and Bayesian analysis?

Third: for comments not referred to, I simply disagree with your conclusions. In most cases finding them some combination of rude, ill-informed, misguided, or simply wrong.

0. Weak, and trivially abused, verification basis.

I address this partially in my post. I can see lots of people white listing microsoft.com or ebay.com or similar. TMDA provides a solution with its tagged addresses, since challenge/response and a white list are not the only tricks up its sleeve. SpamAssassin (for instance) has a white list also, but if a spammer abuses it (as you say they already do), there's no easy way out. With TMDA, I can go to MS, tell them my address changed to a tagged address, and take them off my white list.

Spamassassin's whitelist isn't a "pass all from address", but acts by modifying the score of a message. The autowhitelist feature simply applies an average of recent scores to the score of the current message, weighted to favorable dispensation to avoid false positives. Not perfect, but in general, The Right Thing.

Ultimately, I'd rely on stronger proof such as a valid (and preferably trusted) GPG signature, validated on receipt or as part of filtering.

1. Mistaken interpretation of anti-spam goals

I'm confused. Are you telling me I don't have the goals I do?

I'm telling you that the result of C-R systems, in practice, is not what you claim it is.

3. Privacy violation.

I run TMDA on my own mail server, so this isn't a problem for me. I can't help but wonder if you think that ordinary mail logging is also a privacy violation. Should we lobby for our ISPs to turn off the logs generated by their mail systems?

To a four nines approximation, regular users don't control their mailservers.

TMDA, to work, requires a permanent, persistent, subpoenable, and possibly falsifiable record of those with whom you routinely correspond (often the problem with evidence isn't that which exists, but that which is fabricated). Normal system mail logging isn't perpetual. Typically, few of these logs are retained for more than a transient period (speaking from direct ISP experience).

TMDA raises this issue orders of magnitude over existing email logging.

4. Less effective at greater burden than receiver-side whitelisting.

My own experience is that TMDA is more effective than what I was using before.

Receiver-side whitelisting means: I read at most one mail from a given sender, if the first mail from that sender isn't otherwise tagged as spam. My mail burden (100s to 1000s of messages daily) isn't so great that I've got to step on the toes of every poor sap, ex-girlfriend, potential employer, extended family member, or adoring fan who drops me a line.

My general attitude to C-R advocates WRT outsourcing their whitelists is "surely thou dost protest too much".

5. High type II error (beta).

I don't get this. Maybe I need to read the links. What do you mean by "issues in sender-compliance"?

Maybe you should.

"I don't get this", BTW, seems bloody typical (viz: "I don't know anything about this.") of C-R proponents, many of whom define any message to which a challenge is not responded as a rightful blocking.

I mean: "I'm not going to respond to your goddamned C-R request".

The problem with spam is that each spam message has an incremental, but low, negative cost. Each missed valid mail has a very high, positive cost. Spam might cost me, say, one tenth of a cent, uniformly. A missed mail may cost me a missed communication with a friend, or $100,000 or more, if it's a client or recruiter contact.

Spam-filtering systems cannot have any appreciable type-II error to be generally useful.

6. Potential denial of service. I address this in the article and in an earlier comment. Briefly, this problem is easily found in other aspects of the system, and simply closing an account creates a larger problem than using TMDA on one.

Joe Job. Again. Sample. Closed accounts can't be used for attacks targeted at an arbitrary individual.

7. C-R - C-R deadlock

This is addressed in the article and an earlier comment. Basically, TMDA doesn't have this problem.

Basically, TMDA isn't the only, or even principle, implementation of C-R. See "a well designed system won't do that" above.

8. Potential integration into spam email harvest systems.

I addressed this in the article. Briefly, doing this is already possible and doing it increases the cost of sending spam. The former is no loss, and latter is a win.

Of the various problems introduced by this aspect, a key is social conditioning. People are already told not to reply to opt-out addresses or links, aka "valid email confirmation mechanisms". C-R stands a high probability of being perceived, probably rightfully, as such a system. This reduces C-R compliance, increases type-II error, and reduces overall usability of email.

9. Likely consequences: C-R messages and users blacklisted or spamfiltered

As you suggest, perhaps it can't be helped. It looks a lot like other collateral damage, though. I was blocked by a RBL long before I started using TMDA.

Funny you should mention. First off: yes, it's inevitable. Second: it can't be helped. There is no technical solution to this from within C-R systems. Third, C-R, like RBL, is a shotgun approach. RBLs are useful for assessing possible spamminess of a given post when weighted appropriately. Adaptive filters such as Spamassassin do this, automatically, through automated training against spam corpuses. Blind reliance on RBL, or C-R, or, say dynamically assinged IP mailhost banning (yes, read, or at least glance at, the links), suffers a high false-positive, type-II, error.

10. Mailing list burden.

TMDA doesn't have that problem.

See "a well-designed system" above.

11. Fails to address techno-economic underpinnings of spam.

To bypass TMDA effectively (with intelligent white list exploitation or a challenge auto-responder) raises the cost of sending spam.

Very marginally, and by means readily bypassed.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Can we talk? (3.50 / 4) (#173)
by Kyle on Sat Aug 23, 2003 at 03:22:33 PM EST

"I'm not restricting my discussion to TMDA, but C-R in general." and "Poorly behaved systems are the norm."

So, you're arguing against something I'm not defending. I'll grant you that there are far more poorly behaved C-R systems than there are well-designed C-R systems (like TMDA). They cause problems, one of which (from my perspective) is that people encountering a well-designed system (like TMDA) are prejudiced against it. I can understand other systems giving this one a bad name. I feel fortunate to have been able to discover TMDA without such heavy bad experiences in my past.

Moreso: C-R is effectively narrow content-based filtering reduced to simply the "From:" header of an incoming mail message. Why not availe yourself to the rest of the bloody message, via a comprehensive content and Bayesian analysis?

You can, as previously mentioned, even in the FAQ. I think a combined solution is best overall, but it was not the topic of the article.

Spamassassin's whitelist isn't a "pass all from address", but acts by modifying the score of a message.

If a white list isn't "pass all from address", is it really a white list? As I recall, white listing an address in SpamAssassin adjusted its score so much (-100) that it was pretty well assured of going through. I understand you can adjust that, but I think of a white list as a guarantee of delivery. YMMV.

I agree that a cryptographic signature is a great way to authenticate mail, however the people using things like GnuPG are few and far between. It's a great solution that doesn't solve the problem for me right now.

"To a four nines approximation, regular users don't control their mailservers."

...but they do still have to trust their admins not to invade their privacy. If you're concerned about having a white list laying around, you can use a sender address for anyone who you don't want to appear on your list. (Before now, I didn't see what use they had, so thanks for pointing this out.) Sender addresses work without a challenge, provided the mail is sent from the address that's coded into it. So, I'd generate a sender address for you to use, but other people couldn't use it. It's kind of like storing my white list entry for you in your address book.

I admit, this is not a great solution to the problem. I see your point about having a list of contacts laying around where Bad Guys can find them. If that's a concern for you, by all means, keep away from this software.

The problem with spam is that each spam message has an incremental, but low, negative cost. Each missed valid mail has a very high, positive cost. Spam might cost me, say, one tenth of a cent, uniformly. A missed mail may cost me a missed communication with a friend, or $100,000 or more, if it's a client or recruiter contact.

If your email is worth that much to you, then you have to filter carefully indeed. If you think people won't respond to the challenges, then don't use it. The people you correspond with are different than the ones I talk to, and gauging how they react to this is an important consideration before deploying it.

If it makes any difference, TMDA provides fairly easy access to the pending queue, if you ever want to scan it for legitimate mail that's not being confirmed (what you call "type-II error"). I looked at it for a month, and it was a complete waste of time.

You mention "high type-II error" a lot. Do you have evidence of this? On what basis do you call it high?

Closed accounts can't be used for attacks targeted at an arbitrary individual.

Am I missing something? Just this morning, I got a mail that says, among other things, this:

----- Transcript of session follows -----
553 5.3.0 mihail@bizon.vyatka.ru... No such user in this domain!

This is a result of some spammer putting my domain as the return address on the message sent to mihail@bizon.vyatka.ru. Now, if you want to Joe Job someone, and you can't find a TMDA user to do it, just put their name at the top of some message you send to that address (or any address that doesn't exist).

Joe jobs are a problem, but challenge/response systems are not their real source.

Of the various problems introduced by this aspect, a key is social conditioning. People are already told not to reply to opt-out addresses or links, aka "valid email confirmation mechanisms". C-R stands a high probability of being perceived, probably rightfully, as such a system. This reduces C-R compliance, increases type-II error, and reduces overall usability of email.

So, you're saying that people are told not to confirm their address for fear of giving it to spammers, therefore they won't confirm their messages? An inexperienced user could have this problem, I'll grant you, but I don't think it's hard to distinguish a confirm request which comes from the person you just tried to email vs. a confirm request that comes out of the blue. In the absence of any hard data, my opinion is that this is not a widespread problem. If you have evidence to the contrary, I'm interested. If not, you're welcome to your own opinion (and to decide on how to deal with your mail accordingly).

I still think TMDA is a really good solution, and I intend to keep using it. That having been said, I see now some implications of it that I hadn't considered before, and I appreciate you taking the time to discuss these issues with me.

[ Parent ]

McFly! (1.33 / 3) (#213)
by sllort on Wed Aug 27, 2003 at 02:30:40 PM EST

TMDA, to work, requires a permanent, persistent, subpoenable, and possibly falsifiable record of those with whom you routinely correspond
Are you saying TDMA cellphones allow Verizon to store a list of people you call? Or were you saying that AOL-IM allows AOL to store a list of people you message?

Or were you neglecting the fact that we crossed this hurdle years ago?

Please open up a website for PayPal donations, we can get together and buy you a clue.
--
Warning: On Lawn is a documented liar.
[ Parent ]

Data retention and privacy (4.33 / 3) (#214)
by kmself on Wed Aug 27, 2003 at 06:17:28 PM EST

Well, I work IRL as a SAS programmer, on such databases as healthcare, pharmaceuticals, banking, investments, consumer credit, insurance, and others.

I've also worked as a sysadmin for ISPs handling moderate amounts of mail (~15,000 accounts, ~40,000 messages daily).

In most cases, live-time online transactional data are only retained for a brief period of time, typically three months in banking and credit card data. Phone records, as I understand, have similar retention periods. Problem for the entity archiving this data is that there's simply too much of it to keep, let alone analyze usefully. Data warehousing is largely a matter of first summarizing usage data to reasonable aggregation points. Transaction-level data is lost in the process: it's simply not useful from a business sense.

In the case of ISPs, many rotate their mail logs through their backup/retention schedule, often meaning that total retention is for a few months, or possibly a year on select snapshots. Again, the data storage requirements, backup time, etc., are simply not justifiable for the value of the data. You want to be able to go back through yesterday's, or last week's, or last month's logfiles if a customer calls with a problem. Anything more than two month's old is typically "well, try to note the problem sooner, we don't have the data".

Already, security-concious sites such as Cryptome routinely keep their logs only long enough to solve pressing current administration issues -- otherwise, logs are deleted daily, or more often as required.

There are cases in which data are retained for longer periods of times. Brokerage and healthcare both come to mind. I piss in jars and leave prints to get near the stuff -- the data are highly regulated. Even where long-term data retention is required, much of this is retained offline or near-line, rather than immediately online.

For anyone who's been involved in InterWeb affairs over the past five years, issues of dot-coms suddently deciding that customer / visitor data were a strategic asset should be an immediate concern.

C-R lists differ in that the data must remain immediately online indefinately.

Of course, this is an issue that falls into the general class of "for those people who care about this sort of thing, this is the sort of thing they care about", to misquote Lincoln.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Sir! (1.75 / 4) (#215)
by sllort on Thu Aug 28, 2003 at 10:56:47 AM EST

You have failed to make a coherent argument!

Your AOL "buddy list", which is a list of people you converse with, lasts for at least two years! I have AOL accounts that are two years old that I haven't touched, just checked them and their buddy lists are still intact!

I am not talking about individual message transaction logs, nor is anyone in this conversation. Your straw man is choking on... straw!

Have you ever noticed that your cell phone's directory isn't stored on your cell phone? Buy a new phone, boot it up, and they PUSH your contact list from the server! This list is persistent as long as you're a customer! It is, as the original poster commented, a persistent, subpoenable record of people you converse with, just like AOL buddy lists!  Both are stored server side, both are lists of people you converse with - they are identical in nature to C-R lists!

Let me tell you if the FBI wants to know who you're emailing, they'll hook Carnivore to your uplink at the ISP and sit there munching donuts while they reconstruct your midget porn browsing with Silent Runner.

Of course you rate and reply in the same thread thereby proving you're an idiot, so I shouldn't be biting. But hey, I'm bored! Have you ever stopped to think that maybe someone involved in a dispute has no place judging the quality of the opposition's opinion? The ancient Greeks came up with a system whereby a a group of neutral third peers were used to decide merit in a dispute! This system has lasted so long we even use it in America, but apparently in your neck of the woods the prosecuting attorney gets to decide who's guilty too!

Have fun in Bangladesh or wherever you're from, bucko. I'd down-rate your comment BUT THEN I'D HAVE TO RENOUNCE MY CITIZENSHIP!!! Or maybe I'll just join the War on Terror and prounounce you an enemy combatant. Voltron told me to.
--
Warning: On Lawn is a documented liar.
[ Parent ]

Moderation abuse (3.00 / 2) (#217)
by kmself on Fri Aug 29, 2003 at 11:20:32 PM EST

Yeah, I suppose I don't have a clue what the K5 moderation system is for, or how it was designed. People like me should just be banned from the Interweb, right?


--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Wow (3.60 / 5) (#222)
by sllort on Tue Sep 02, 2003 at 11:10:58 AM EST

Knowing that one of the people behind K5 moderation thinks it's cool to down-rate anyone who proves that your argument is BS sure explains a lot.
--
Warning: On Lawn is a documented liar.
[ Parent ]
I moderate as I see fit (5.00 / 2) (#224)
by kmself on Thu Sep 04, 2003 at 04:19:51 AM EST

If you disagree, moderate accordingly. This includes overmoderating for corrective value, if you wish. If you're interested in educating yourself, rather than trolling, you'll follow links or do research yourself (Meatball Wiki has a long discussion) on what thoughts went into the system and what protections exist.

If it restores any of your faith in K5, the creator of the moderation system doesn't have trusted status within the system.

And at the end of the day, it's still just a fucking computer program. Get a life.

IHBT, IHL, HAND.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

That does not change the fact (none / 0) (#227)
by sllort on Mon Dec 08, 2003 at 12:39:40 PM EST

That both cell phones and Instant Messenging services carry a persistent log of who you converse with, just as the C-R systems you're complaining about do. Therefore your argument is bogus.

And all your credentials do nothing to change this fact.
--
Warning: On Lawn is a documented liar.
[ Parent ]

C-R as email harvest (5.00 / 2) (#178)
by kmself on Sun Aug 24, 2003 at 02:53:47 AM EST

I'm realizing there's a disconnect between on regarding the use of C-R as a spammer's email harvesting system.

The problem isn't one that applies to C-R system users, specifically, but to all email users.

Conditioning to respond to C-R requests means that a spoofed C-R challenge would elicit a request (I don't remember sending mail to foo, but I might have, I'll do the Nice Thing and validate it...). Presto, another confirmed live email address.

As the casual user won't be able to distinguish valid from invalid C-R posts, let alone various C-R systems from one another (see "well-designed systems" in prior responses).

Note again, this isn't an attack against C-R users. It's an attack against any email user.

This is another reason I expect compliance rates, and popular advice, to recommend against responding to C-R challenges.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Harvesting made easy. (2.50 / 2) (#181)
by Kyle on Sun Aug 24, 2003 at 02:46:16 PM EST

So, a spammer sends something that looks like a challenge, and the hapless user responds, thus confirming that they read their mail. Better yet, pretend to be a system that uses web confirmations (TMDA doesn't), and trick them into following a link with untold misery at its end.

The problem is made worse when lots of people have a C-R system because people will see more challenges and pay less attention to them. It's made worse still by diversity of C-R systems since it's harder to spot a forgery when you see unfamiliar (but legitimate) challenges regularly.

That having been said, I doubt this will be huge. Few people send messages to new addresses often, so challenges will still be rare for most (allowing them to pay more attention to them).

Also, if the spammers can't guess your address, they can't send you the fake challenge. If they can guess your address (maybe because they already have it), then you're already getting spam anyway. Confirming for them a speculated address is a service to the spammer, but this probing itself is harder work than what spammers do now to get lists, driving up the cost of spamming. A spammer wouldn't be interested in culling their mailing list until it's cheaper to probe once in a while than to send to bogus addresses all the time.

If you want to detect such an attack, it shouldn't be too hard. The easiest way (especially for end users) is getting a challenge without sending a mail. (The other side of this is a legitimate challenge could be delayed by a disconnected mail server.) Also, TMDA (by default) includes the message that it's challenging in the challenge, so the user can see what it's talking about. I'd consider a challenge pretty broken indeed if it did not include some reminder to the challenge receiver. An alert user could use those clues to decide if the challenge is real.

[ Parent ]

Death spiral (5.00 / 2) (#192)
by kmself on Sun Aug 24, 2003 at 10:58:31 PM EST

That having been said, I doubt this will be huge. Few people send messages to new addresses often, so challenges will still be rare for most (allowing them to pay more attention to them).

Problem with that rationale is: if most C-R challenges are sent by spammers, most people will start ignoring C-R challenges.

The memetic pollution is two-way. C-R recipients start ignoring C-R mail. C-R users can rely less and less on challenges being responded to. Vicious circle.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Not dead yet! (1.50 / 2) (#198)
by Kyle on Mon Aug 25, 2003 at 08:38:56 AM EST

When C/R becomes prevalent, perhaps what you describe will become a problem. Someday, I imagine, the only good way to filter mail will be with digital signatures. Until the spammers get that smart, TMDA will be a good way to keep them out of your mailbox while letting the rest of humanity in.

[ Parent ]

Bingo (3.66 / 3) (#199)
by PigleT on Mon Aug 25, 2003 at 12:24:04 PM EST

Oh boy, that's a wonderful summary of everything I've thought of as being a reason *not* to use TMDA, and more besides.

I agree entirely; it's immoral to shift the workload  back onto the sender, the risks of third-party back-scatter are waay too great - I've seen the mail side of a mail2news gateway passing TMDA challenges onto the newsgroup, which is probably the definition of spam; I truly hate to think what it's doing to everyone in the current Sobig.F virus wars.

In short, it doesn't scale to the extent that everyone could use such a thing, therefore really shouldn't be used at all.
~Tim -- We stood in the moonlight and the river flowed
[ Parent ]

Welcome to SpamCop (5.00 / 1) (#207)
by kmself on Tue Aug 26, 2003 at 04:32:24 PM EST

Kyle (he's a nice guy, just...misguided) and I have been correponding OOB, he actually sent me this.

A TMDA user found himself added to SpamCop as a spam source. http://mla.libertine.org/tmda-users/2003-08/msg00171.html.

The SpamCop report itself is here.

Valid report? IMO, yes. TMDA, one of the better designed C-R systems, trashes the world in the name of protecting the inboxes of its users.

To date, I've had more C-R challenges sent to me on the basis of spoofed mail than legitimate first-time posts.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

C-R: AS for people who can't do maths or ethics (4.50 / 2) (#218)
by kmself on Sat Aug 30, 2003 at 12:53:04 AM EST

Emphasizing objection #2, let's look at what sorts of numbers we come up with if a small number of people are using C-R systems.

The conclusion is inevitable: challenge-response is anti-spam for people who cannot do maths or ethics.

The numbers here are reasonable facsimilies of reality, though the ranges many vary. Go with the numbers, then adjust as necessary.

We're assuming for this model:

  • 600 m mailboxes reaching humans.
  • 40% of email is spam.
  • Average user receives 60 emails daily (25 spam, 35 ham). That's 7 trillion emails annually.
  • And for the purposes of analysis, 1% of users have a C-R system.
  • Each spam message spoofs a valid email address.

On a given day, 6m users receive 150m spam messages, and send 150m C-R challenges.

At this rate, any given email users receives a C-R challenge based on a spoofed email, once every four days.

Legitimate challenges might occur once a month or so. The probability that a given challenge is spam is 8:1, or 0.89.

If the penetration of C-R usage rises to 25% of email users, then rather than receiving 0.25 spoofed challenges per day, I'm getting 6.25. Or 20% of my spam is C-R challenges.

Of course, this situation is not fully accurate. The amount of spam that spoofs sender with actual addresses is probably smaller -- I'll assume 10-25% (though virus mail probably ups this again). It's also possible that a given spam mailing tends to use only one, or a small set, of sender addresses. Which means that odds of my getting slammed are significantly smaller, but when I do...the problem isn't 0.25 C-Rs in a day, or even 6.25, but 150,000,000 of them. I'll be off the 'Net for weeks.

And the problem gets worse the more C-R scales. Which means that people who think that C-R is a good idea now are contributing a huge reeducation problem down the road when we finally do realize that C-R doesn't work, messages are ignored, and the vast majority of C-R challenges are sent to the wrong person, possibly at very significant harm.

This is the harm in the proposal.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

SMTP considered harmful (3.50 / 2) (#219)
by Kyle on Sat Aug 30, 2003 at 09:43:14 AM EST

Properly functioning SMTP servers produce far more of this stuff than C-R systems. See my diary where I discuss the real world effectiveness of my TMDA installation.

In three weeks, I got well over 5000 spurious bounce messages from well-meaning SMTP servers. In over a month, I got about 100 challenges. These are all responses to emails I did not send.

I suspect that SMTP servers will always be more prevalent than C-R systems. When SMTP is fixed such that it does not bounce to people who did not send, C-R systems will be fixed too. In the meantime, TMDA is a very effective way to stop spam, and its impact on the network is no worse than a server receiving mail for a user who doesn't exist.

[ Parent ]

Already covered. (5.00 / 1) (#220)
by kmself on Sat Aug 30, 2003 at 09:49:02 PM EST

See the debian-devel / BTS discussion.

First: because another system is broken doesn't mean that it is appropriate for your system to be broken in the same manner.

Second: SMTP nondelivery messages are generated by the mail system after an earnest, best-faith effort has been made at local delivery. The message is undeliverable. As a courtesy to the (possible) sender, a notice of this is sent. By contrast, a C-R challenge is only sent on a successful delivery to which the recipient can't be bothered to assess whether or not they are interested in receiving the message, and have offloaded this workload to an unathenticated and likely inappropriate third party.

Third: As you note, there are somewhat more email addresses sitting behind SMTP servers than are behind C-R systems. One might safely assume that 100% of email addresses are behind SMTP servers. The total usage of C-R systems is likely far, far less -- less even than a small fraction of 1%. You don't quantify the number of challenges you received in your diary article, but it would be at least 0.9% (one in 109 messages), I suspect it's several percent. Which indicates that C-R systems are generating spam in a proportion several orders of magnitude greater than their representation in the general population. And as my model above shows, the situation only gets worse.

Fourth: the problem of SMTP responses to spoofed senders can be addressed in the same way that C-R message generation can: by first dealing with messages at SMTP time, by eliminating virally-generated mails, by eliminating spam, and, as a last resort, sending a notification to the small subset of misdelivered mails which cannot be intercepted at SMTP time, and are not clearly viral in origin, or spam. Note that this also allows for automated notification of upstream senders at SMTP time of problems with viral or spam mail through a given sending host.

In short: Yes, SMTP delivery and transport agents generate misdelivered notification messages. You're using this to dodge the question of why C-R and TMDA shouldn't. THe real question is: how do you fix both systems so that they don't do this?

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

If one standard is good, two must be better. (2.00 / 1) (#221)
by Kyle on Mon Sep 01, 2003 at 10:41:34 PM EST

See the debian-devel / BTS discussion.

I did. It made me wonder if you'd filed grave bugs also against Sendmail, Exim, Postfix, etc.

First: because another system is broken doesn't mean that it is appropriate for your system to be broken in the same manner.

This is entirely true. The question I'm asking you is, why are you comfortable with SMTP servers throwing 5000 misplaced emails at me, but C-R systems sending me 50 or so is bad, bad, bad? Answers below, I guess...

Second: SMTP nondelivery messages are generated by the mail system after an earnest, best-faith effort has been made at local delivery. The message is undeliverable. As a courtesy to the (possible) sender, a notice of this is sent. By contrast, a C-R challenge is only sent on a successful delivery to which the recipient can't be bothered to assess whether or not they are interested in receiving the message, and have offloaded this workload to an unathenticated and likely inappropriate third party.

So, Sendmail is welcome to throw crap at me because it has different procedures for deciding when to do it? I think when deciding this issue, the intentions of the software involved is not relevant. Automatic responses are OK, or they're not. They follow protocol or they don't.

Moreover, I can't help but think that a challenge is a type of delivery failure worthy of notification to the sender. If you're not on my white list, the message is undeliverable. As a courtesy to the (possible) sender, a notice of this is sent along with information on how to get the message delivered.

Contrast that to a normal filter setup. My mail is not delivered, but I have no idea that it even happened, let alone why. That's better? It's not better for the sender or the recipient. It's only better for the third party, who's only involved when the sender is misusing the system.

In other words, spammers are trying to hurt some recipient, but the recipient is (maybe) deflecting it to the third party (chosen by the spammer). It's like the guy who plugs the drain in his basement when the sewers start to back up. He shouldn't be allowed to do that because all the other basements in town get backed up that much more? You may think that way, and I respect that, but I don't think that way.

Third: ... Which indicates that C-R systems are generating spam in a proportion several orders of magnitude greater than their representation in the general population. And as my model above shows, the situation only gets worse.

Would UCE be OK if there were only one a year? I, for one, would probably not bother with anti-spam software in that case, but I wouldn't look at the numbers and use that to declare when a certain type of email is bad.

Fourth: the problem of SMTP responses to spoofed senders can be addressed in the same way that C-R message generation can: by first dealing with messages at SMTP time, by eliminating virally-generated mails, by eliminating spam, and, as a last resort, sending a notification to the small subset of misdelivered mails which cannot be intercepted at SMTP time, and are not clearly viral in origin, or spam. Note that this also allows for automated notification of upstream senders at SMTP time of problems with viral or spam mail through a given sending host.

The problem with issuing challenges at SMTP time is that they'd be lost in the noise of a bounce message when the user saw them, and that one line of challenge text is hardly enough to convey the message. I think you'd have a lot more people not responding to the challenge because they don't realize it's not a failure notice.

Other than that, I love the idea!

I agree that filtering out obvious spam, viruses, or other flotsam that may come up is a great idea. I do that at home. That having been said, it's outside the scope of a C-R system to do that. I'd much rather have TMDA specialize in the C-R part of the email handling job and do the rest with programs specializing in those tasks.

In short: Yes, SMTP delivery and transport agents generate misdelivered notification messages. You're using this to dodge the question of why C-R and TMDA shouldn't. THe real question is: how do you fix both systems so that they don't do this?

Indeed that is the real question. The bonus question is, "is it reasonable to hold TMDA responsible for SMTP's lack of authentication, but not Sendmail?" Are list management packages broken by design because they reply automatically to email addresses that everyone knows can be faked? If not, why is TMDA?

You try to draw distinctions between these cases, but I don't see them. List servers are trying to impose some minimal authentication on messages. So is TMDA.

[ Parent ]

Khendon's law (5.00 / 1) (#223)
by kmself on Thu Sep 04, 2003 at 03:01:28 AM EST

You haven't raised a single new point in your post. You've rearranged several slightly. This will be my final comment on the subject barring novel statements.

See the debian-devel / BTS discussion.

I did. It made me wonder if you'd filed grave bugs also against Sendmail, Exim, Postfix, etc.

Don't insult my intelligence or patience by repeating arguments already addressed. May I reaquaint you with Khendon's law? Please provide the courtesy of reading a post fully, composing your response, and avoiding redundancies. I do this for you.

First: because another system is broken doesn't mean that it is appropriate for your system to be broken in the same manner.

This is entirely true. The question I'm asking you is, why are you comfortable with SMTP servers throwing 5000 misplaced emails at me, but C-R systems sending me 50 or so is bad, bad, bad? Answers below, I guess...

Discussed on BTS. Khendon's law invoked.

Second: SMTP nondelivery messages are generated by the mail system after an earnest, best-faith effort has been made at local delivery. The message is undeliverable. As a courtesy to the (possible) sender, a notice of this is sent. By contrast, a C-R challenge is only sent on a successful delivery to which the recipient can't be bothered to assess whether or not they are interested in receiving the message, and have offloaded this workload to an unathenticated and likely inappropriate third party.

So, Sendmail is welcome to throw crap at me because it has different procedures for deciding when to do it? I think when deciding this issue, the intentions of the software involved is not relevant. Automatic responses are OK, or they're not. They follow protocol or they don't.

See: John Goerzen's post.

Sendmail isn't offloading its workload to you. It's not threatening to withhold delivery because you're not being its biatch. It's telling you it thinks it got a message from you, and that it can't deliver it. You can safely disregard this notice without invoking further penalty.

Moreover, I can't help but think that a challenge is a type of delivery failure

...at the discretion of the recipient, not the sender. Recipient's problem, as recipient can resolve the conflict by changing their behavior. Next?

Contrast that to a normal filter setup. My mail is not delivered, but I have no idea that it even happened, let alone why. That's better? It's not better for the sender or the recipient. It's only better for the third party, who's only involved when the sender is misusing the system.

I beg your pardon?

SMPT is a non-guaranteed delivery system. If you want a guarantee of receipt, you're going to have to arrange for some form of handshake protocol, be it programmatic or social. This is hardly a sufficient excuse for invoking the ills of C-R on thousands of innocents, as is documented in the tmda-user mailing list.

Third: ... Which indicates that C-R systems are generating spam in a proportion several orders of magnitude greater than their representation in the general population. And as my model above shows, the situation only gets worse.

Would UCE be OK if there were only one a year? I, for one, would probably not bother with anti-spam software in that case, but I wouldn't look at the numbers and use that to declare when a certain type of email is bad.

If you're asking "is this a matter of magnitude", then the answer is yes. If you're insisting that C-R is somehow harmless -- well, what's your model for demonstrating that C-R will only result in a single inappropriately delivered message per email recipient worldwide?

That's not an idle question: if you could fix C-R, and not that that's C-R, not merely TMDA, such that this would be the case, my objections would fall aside. I see no practical way in which this can be accomplished, however.

You're dodging the issue again, Kyle.

Fourth: the problem of SMTP responses to spoofed senders can be addressed in the same way that C-R message generation can: by first dealing with messages at SMTP time, by eliminating virally-generated mails, by eliminating spam, and, as a last resort, sending a notification to the small subset of misdelivered mails which cannot be intercepted at SMTP time, and are not clearly viral in origin, or spam. Note that this also allows for automated notification of upstream senders at SMTP time of problems with viral or spam mail through a given sending host.

The problem with issuing challenges at SMTP time...

That's not what I said. I said "dealing with messages at SMTP time" -- eliminated viral mail, spam, etc. The bulk of C-R false challenges likely result from spam and virus mail. There are existing, freely available, free software solutions for both these problems.

Other than that, I love the idea!

Good.

I agree that filtering out obvious spam, viruses, or other flotsam that may come up is a great idea. I do that at home. That having been said, it's outside the scope of a C-R system to do that.

While it may be outside the software's specific design, I maintain that it be mandetory for C-R systems to be structured such that challenges are not issued for any mail that can be tagged by existing, free software, functionally effective, AV/AS filtering software.

My aim in the Debian bugreport is that structured templates and package dependencies require installation of AV and AS software, and that templates for filtering not enable C-R functionality unless AV and AS are present, and applied.

In the tradition of free software, it's possible to extend sytems by embedding functionality. As Spamassassin (for example) is implemented in Perl, it would be possible to incorporate it in TMDA via Python-Perl embedding.

I'd much rather have TMDA specialize in the C-R part of the email handling job and do the rest with programs specializing in those tasks.

If C-R doesn't function appropriately, I will continue to strongly discourage its use.

Given the "well-designed system" objection I deal with in my C-R essay, I strongly suspect that even if some C-R systems incorporate multi-level filters, and only challenge messages which have passed multiple levels of filtering, that I'd have to oppose the concept on grounds of general, not specific, design failures.

As TMDA is designed as, essentially, a mailbox watchdog, I'm suprised at your reluctance to allow its extension in forms which would make it truly effective.

And I will refuse to answer challenges on principle in any event.

In short: Yes, SMTP delivery and transport agents generate misdelivered notification messages. You're using this to dodge the question of why C-R and TMDA shouldn't.

No, I've answered this question in my previous post, and again here.

In large part, it's that C-R, and TMDA, are designed to generate messages to unauthorized recipients as a principle goal, and to bludgeon the recipient with threatened nonaction as a result. This is a misplaced burden.

It secondarially leads to a breakdown in social conditioning that ultimately undermines C-R's benefits.

In other words: C-R misplaces workload, and undermines its own objectives. By design.

The real question is: how do you fix both systems so that they don't do this?

MTAs should not send bounce replies to spam or viral mail either. Yes, currently, many do. In many cases, this is avoidable, and the bounces occuring today come from poorly configured systems. These should be properly configured.

Indeed that is the real question. The bonus question is, "is it reasonable to hold TMDA responsible for SMTP's lack of authentication,

Yes. See "discretion of recipient" above.

...but not Sendmail?"

Different problem. Different goals. Somewhat addressable at SMTP time during interactive deliveries. Somewhat confounded by issues of secondary MXs and batch spooling and fetching (e.g.: fetchmail). Largely ameliorable.

Are list management packages broken by design because they reply automatically to email addresses that everyone knows can be faked? If not, why is TMDA?

Misconfigured, yes. Broken by design, no.

The "broken by design" component of C-R is this: your inability to determine the legitimacy of the sender of a particular email in your receipt is your problem. Not mine. If you pass that job off to me, I'll refuse to perform it for you. That is your problem. Not mine.

--
Karsten M. Self
SCO -- backgrounder on Caldera/SCO vs IBM
Support the EFF!!
There is no K5 cabal.
[ Parent ]

Mail delivery is everyone's problem. (1.00 / 1) (#225)
by Kyle on Thu Sep 04, 2003 at 11:50:31 AM EST

Your main objection seems to be that TMDA (and C-R in general) offloads the workload (to the sender) of deciding what to deliver. I think this is a basic difference of opinion, and I expect no progress on this issue. I think the sewage in the basement analogy is a good one, and it's obvious that some people would disagree on that.

To me, the fact that sendmail is responding to declare defeat and TMDA is responding to offer victory is a small difference, and one that favors TMDA at that. You may be happier if you consider challenges to be simple delivery failure notifications and treat them accordingly (refusing to answer challenges "on principle" is effectively just that).

I think that integrating anti-spam and anti-virus software into C-R installations (to cut down on spurious challenges) is a good idea, but I don't see a reason to integrate it into the C-R software itself. Making TMDA a "mailbox watchdog" is, I think, overgeneralizing its role. Basicaly, I think that's procmail's job.

what's your model for demonstrating that C-R will only result in a single inappropriately delivered message per email recipient worldwide?

I don't have one. Since I consider challenges to be strongly akin to bounces (and we don't limit them "one per customer"), I don't consider this a problem.

As for the social conditioning problem, I think we covered that.

[ Parent ]

*sigh* (4.66 / 3) (#182)
by DavidTC on Sun Aug 24, 2003 at 03:44:42 PM EST

Spammers forge the address of an innocent victim when they spam, so that bystander will get a huge pile of challenges.

That is unfortunate, but it's no different than:

Vacation messages.
Bounce messages.
List subscription confirmation messages.
List moderation notices ("Your message is being held pending moderator approval").
That last sounds the most like the challenges we're talking about. Should mailing lists stop doing that because of the innocents affected? (Incidentally, TMDA is designed not to try to confirm automated messages like the above.)

My point is that given you have to deal with bogus automatic responses generated by spammers already, this is not such a larger problem. The responses generated by TMDA are made to look automatic to existing software. In these terms, a TMDA user is no more a burden to the network than a user who abandoned an address (perhaps because it was being spammed as heavily as the user who switched to TMDA).

The problem is that those software are causing serious problems, already. Or did the entire SoBig mess pass you by? The bounces and automated 'You have a virus' messages are much much more annoying to serious mail adminstrators than the actual virus.

Bounces are the Wrong Thing, and this has been recognized for several years now. Automated replys are the Wrong Thing, period.

Not a day goes by where some idiot doesn't send a vacation message to some mailing.

TMDA may, indeed, be some perfect solution to spam, but it solves the problem by offloading it onto other, completely random people. The problem isn't the burden you place on people sending you messages, you can burden them all you want. The problem is the mindbogglying idiotic assumption that the 'sender' of an email message has the slightest thing to do with the actual message.

Mail list messages are only tolerated because they stop spam to the person the message is sent to. I can put up with a few lists emailing me if that stops people from forge-subscribing me. A forged confirmation bomb is much better than a forged subscription bomb.

Whereas C/R does not stop spam to the person the C/R message is sent to. There is no benefit to me from your use of C/R, especially if some spammer forged my email address and I have no idea who the fuck you even are.

And, yes, I've gotten wayward C/R messages, although apparently from forging viruses instead of forging spammers. And, no, that doesn't make it any better...it was sent in bulk to everyone who's address was forged, it's unsolicted, it's email...that's spam, people. Anyone using C/R to stop spammers has managed a cliche few manage, you've turned into your enemy in your attempt to defeat him.

And, yes, people have already started blocking mail servers who use C/R.

-David T. C.
Yes, my email address is real.

One more thing. (5.00 / 1) (#183)
by DavidTC on Sun Aug 24, 2003 at 04:03:07 PM EST

I'd like to emphasize I have no problem in theory to making mail senders jump through hoops if you want, although be aware that will cause mail lossage. The problem is forgery handling.

And, in fact, there's a perfectly functional method of doing C/R...sending C/R messages as SMTP reject messages, sending a '550 Must answer challenge at http://example.com/blah?sender=whoever' to the sending SMTP client during the transaction. The SMTP client at the other end is supposed to present this to the user.

Sadly, no one seems to be write C/R software that does that.

-David T. C.
Yes, my email address is real.
[ Parent ]

I like that, but... (4.00 / 1) (#186)
by Kyle on Sun Aug 24, 2003 at 04:26:31 PM EST

And, in fact, there's a perfectly functional method of doing C/R...sending C/R messages as SMTP reject messages, sending a '550 Must answer challenge at http://example.com/blah?sender=whoever' to the sending SMTP client during the transaction. The SMTP client at the other end is supposed to present this to the user.

Twenty minutes ago you said "Automated replies are the wrong thing, period." Why is generating this bounce message with the challenge OK, but generating a (more readable and informative) challenge is not OK?

I doubt that message would be read by the typical user. I regularly talk to people who "got a bounce message" and don't know why. They're confused by the format when I ask them to look for the reason it failed. They'd miss the challenge completely.

I like the way your proposal uses existing facilities, but I don't see why it can't be (ab)used by forgers the same way as any other challenge/response installation.

[ Parent ]

550 isn't bouncing (5.00 / 1) (#190)
by martingale on Sun Aug 24, 2003 at 10:02:51 PM EST

550 means "requested action not taken" and is a response code during the SMTP transaction. The mail simply isn't accepted, unlike a bounce which accepts the mail but later finds it cannot be delivered and resends it.

[ Parent ]
Automatic replies considered harmful (3.00 / 1) (#184)
by Kyle on Sun Aug 24, 2003 at 04:04:12 PM EST

Bounces are the Wrong Thing, and this has been recognized for several years now. Automated replys are the Wrong Thing, period.

I hadn't heard that bounces are the Wrong Thing. Can you point to someplace discussing that? If you really believe that, then obviously challenge/response systems are Wrong as well. In any case, I can't imagine automatic responses stopping any time soon. It's amusing to hear that in an anti-spam discussion since virtually every abuse@* address I've emailed has sent me an automatic response.

it was sent in bulk to everyone who's address was forged, it's unsolicted, it's email...that's spam, people. Anyone using C/R to stop spammers has managed a cliche few manage, you've turned into your enemy in your attempt to defeat him.

Does this mean that anyone running a mail server which sends bounces is a spammer also? Are there actually mail servers configured not to send error messages?

[ Parent ]

Bounces vs. rejects (5.00 / 2) (#185)
by DavidTC on Sun Aug 24, 2003 at 04:23:39 PM EST

Bounces are seperate email messages generated by the server and sent to someone. Most modern server reject the message during the transaction, and the sending SMTP server generates the messages that the user eventually sees.

Not only does this reduce message traffic, but it insures that only the actual sender gets the message.

Of course, that doesn't always help if a spammer is using an open relay...the open relay will see the rejection, and happily compose a message from the rejection and mail it off to the 'From' address. But there are almost no open relays left in unblocked IP space. (Don't confuse them with open proxies, which are just SOCK proxies, and have no mail server logic in them, and thus can't send bounces.)

If you want to see the difference, send a random message to a made up address at AOL, vs. a made up address at hotmail. The AOL address will accept the message then bounce it, and the Hotmail address will just reject the message, leaving your server to generate the message that eventually gets downloaded by your mail client, hopfully using the reject string that Hotmail sent. Which is where any C/R message belongs, BTW, in the reject message.

And, yes, AOL is taking a lot of flack for this, especially with SoBig forging addresses left and right. If they weren't the eight hundred pound gorilla of the internet, they'd be blacklisted various places, and as it is people have start blocking bounces from them.

Automated after-the-fact messages have been universally declared Wrong. If you don't want an email, you either reject it during the SMTP transaction, or you drop it on the floor without telling anyone. Sending it back to the 'sender' doesn't accomplish a damn thing, because almost all the email you don't want was lying in the first place.

And almost all C/R software is starting off with the assumption that sending automated messages back to the 'sender' is okay behavior. It was fine five years ago, when everyone wasn't forging, but it's not fine now.

-David T. C.
Yes, my email address is real.
[ Parent ]

Gah, I need to stop replying to my own posts. (4.00 / 1) (#187)
by DavidTC on Sun Aug 24, 2003 at 04:34:54 PM EST

AOL has apparently fixed their system, I just got this when I tried the trick, which is obviously a reject:
----------
This is the Postfix program at host [my server was here].

I'm sorry to have to inform you that the message returned below could not be delivered to one or more destinations.

For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can delete your own text from the message returned below.

            The Postfix program

<asdfsdafafasdfa22sdas@aol.com>: host mailin-01.mx.aol.com[152.163.224.26]
    said: 550 MAILBOX NOT FOUND (in reply to RCPT TO command)

Reporting-MTA: dns; [my server was here]
X-Postfix-Queue-ID: 9C9FB519334
X-Postfix-Sender: rfc822; [My address was here]
Arrival-Date: Sun, 24 Aug 2003 13:20:05 -0700 (PDT)

Final-Recipient: rfc822; asdfsdafafasdfa22sdas@aol.com
Action: failed
Status: 5.0.0
Diagnostic-Code: X-Postfix; host mailin-01.mx.aol.com[152.163.224.26] said: 550
    MAILBOX NOT FOUND (in reply to RCPT TO command)
-------------

Where that MAILBOX NOT FOUND is, that's what I'm talking about for a URL to operate a C/R system.


-David T. C.
Yes, my email address is real.
[ Parent ]

You answered my question, even before I posted it! (4.00 / 1) (#188)
by Kyle on Sun Aug 24, 2003 at 05:08:12 PM EST

Thanks for the explanation of bounces vs. rejections. I guess I haven't had quite enough time with my head stuck in a mail server.

You pointed out already my first thought: open relays. Here are some others:

  • As I said, abuse@* automatically replies.
  • I get lots of bounce messages today (well, it's only been 80 so far today, but you know what I mean). Obviously, this wisdom hasn't reached all corners of the net yet (though this certainly does not mean it's not wise).
  • I like rejections vs. bounces for the reasons you describe, but I think that bounces are still useful. I don't know how I'd find a mail loop if not for the bounce it generates, and I can't expect the first server that sees the message to detect the loop.
  • I seem to recall qmail doesn't know enough during the SMTP transaction to know which emails to reject. It accepts pretty much anything and bounces it later. There are problems with this approach beyond the inability to reject mail, but the counter argument goes that it's more secure that way.

In short, I don't expect automated replies to go away any time soon. That having been said, the idea that "all automated replies are bad" appears sound on the surface. I just think that automated replies are too useful in too many situations to be discarded completely.

[ Parent ]

TMDA shits me (4.00 / 2) (#191)
by seeS on Sun Aug 24, 2003 at 10:03:41 PM EST

If I get one of those reponses unless I *really* want to talk to the person I don't bother.
--
Where's a policeman when you need one to blame the World Wide Web?
challange spams (4.00 / 1) (#195)
by dimaq on Mon Aug 25, 2003 at 04:10:09 AM EST

this was probably mentioned by someone, but if the challange is allowed to get through 'immediately' (and you'd want that), all the spammers with simply switch to sending fake challanges with their spam included.

This could be fixed with... (4.00 / 1) (#196)
by blight on Mon Aug 25, 2003 at 05:25:31 AM EST

a short random token in every new email sent which, if returned in the reply, allows it to pass through without any filtering.

If spam starts coming in with one of the token, simply revoke the token and possibly take some other action based on who originally got the token from you.

The token could be included in the message headers by email agents that support it. Would propably make sense to include the token in the message body to make it easier for people with older email clients to work with it.

[ Parent ]

SPF: verified From adresses (5.00 / 1) (#226)
by joostje on Fri Sep 19, 2003 at 12:10:21 PM EST

Haven't seen SPF mentioned jet.

With SPF, you (domain owner) specify (in DNS) what IP adresses may claim to send mail claiming to come From: your domain.

This would solve the C-R problems.

(Yes, I know I'm late, but well).

TMDA Ends Spam | 227 comments (200 topical, 27 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!