Handling Anti-Spam Systems When You Aren't Spamming? 59
"Many large ISPs are implementing anti-spam filters based on how many emails they receive from a single sender to many of their clients (thinking that if they get over five mails in a few seconds, they must be bulk-mail spammers, and therefore block the rest of them), but this is hurting the delivery of services like ours. Worse still is that there is typically no error message returned to us - the emails simply get dropped, much like a standard packet-filter firewall works. Then we have clients wondering why they didn't get their expected message.
Sometimes, ISPs will add us to their "white" lists (as opposed to "black" lists of known spammers), which fixes the problem, but only for that one ISP.
(I find it ironic that the email system was designed to be quite reliable, so that you could send a message and have reasonable confidence that it got to its intended recipient, and yet we're now moving away from this in the effort to fight spam.)
Now I know we don't want to tell spammers how they can get around the anti-spam filters, but I'm wondering how have others fought the anti-spam problem with their mailing lists?"
Re:Opt in? (Score:1, Insightful)
Re:As they say in Texas (Score:2)
Come the revolution, they'll be the first up against the wall -- someday Denial of Service will be illegal, and then they'll get theirs.
Play the game, but don't go too far. (Score:2, Informative)
I don't know how you are managing your newsletter, but eGroups doesn't seem to have too many problems with that; Either they know how to get through (more probable), or everyone makes an allowance for an egroups address (less probable). Either way, if all else fails consider using egroups or a professional service that works (Never tried myself and am not affiliated with, but I hear whatcounts [whatcounts.com] is good.)
Re:Play the game, but don't go too far. (Score:1, Insightful)
Re:Play the game, but don't go too far. (Score:2)
Re:Play the game, but don't go too far. (Score:1)
Stupid idea (Score:2, Insightful)
That could probably go down as the most stupid idea I've heard so far this year. All this 'monitoring' is sounding way too authoritarian to me.
In the majority of cases, it should be the individual's responsibility to sort mail, not the ISPs. Would you like it if USPS decided to go through your mail throwing away whatever it thought was 'unsolicited'? You bet your ass you wouldn't. How about if they suggested 'looking through your outgoing mail' to find out what you were expecting to receive? If people like you were taken seriously, it'd be like the Third Reich.
I do not want anyone reading or filtering my mail except myself! If you want to be nannied, that's your choice, and you can go use AOL or whatever, but we don't want the majority of ISPs controlling mail delivery in this way. Even if their intentions are good, 'proper' e-mail could easily get thrown away, and worse.. if laws were passed that allowed governments to control ISPs in some way, they'd have a system already in place to 'control' mail delivery. No thanks!
The answer to this question is that any freedom loving citizen should be filtering their own mail and not relying on a nanny state to sort it out for them.
Re:Stupid idea (Score:1)
Re:Stupid idea (Score:2)
(Although, if, for one week, ISPs and Universities stopped blocking spam, it would get to the top of the political agenda really fast...)
Re:Stupid idea (Score:1)
If it was made illegal to be outside between the hours of 10pm and 6am, crime at night would plummet! However, who wants to live in a shitty world like that? The government is all too ready to take away our freedoms, let's not give any of them away by choice.. even if it does save some spam.
I'd rather run the risk of receiving anthrax than to have someone open all of my mail to 'see what's inside'.. if you disagree, then whoa, I hope you don't vote.
Re:Stupid idea (Score:1)
Maybe this already happens, so it really doesn't matter, because all of the spam that we get is not from these servers.
Excuse my thinking out loud
Re:Stupid idea (Score:1)
These guys attack spam at the source
Re:Spam (Score:1)
I'm still trying to get over most ISPs blocking relayings about 4 years ago. I liked being able to send and receipt email from one POP account ... thanks to bastards like this, I now have two options:
1: Read and send from each account.
2: Forward all mail to one account, and only send mail from there. If I need to email from another account, log into that account and mail from it.
I use option 2.. however, it's interesting to note that my main email address is three states away and requires a long distance phone call to send from it with a POP email program.
Who? (Score:1)
I am not aware of any ISP that filter SPAM based upon multiple emails from the same source. That seems pretty stupid to me. Are you sure you're not using an open relay that has been black listed?
Re:Who? (Score:3, Interesting)
More importantly it's a largely waste of time, because we have bounced precisely *zero* emails because of this filter. Obviously the spammers have gotten wise to this filtration method and have worked around it (it's really old after all), which rather makes the whole point of this discussion redundant, doesn't it? ;)
Re:Who? (Score:2)
Yahoo, to pick one example of an email provider, if not an ISP, exactly. If a server sends more than a certain number of emails to yahoo addresses within a certain period of time (I don't know what the specific values are), yahoo will automatically stop accepting mail from that server.
Like some ISPs, yahoo maintains a "white list" of servers that will be excepted from this rule. For an email provider the size of yahoo, this actually makes a lot of sense: there are only a small number of people who will fail the "too much mail too quickly" test for legitimate reasons (other big email providers, for example), so it's easier to work with the small number of exceptions.
I have worked for an email list management company that sends out several million messages per day; yahoo took a look at the company's subscription processes and the messages being sent, decided that their mail was okay, and added them to the white list. No one at the company really minded having to make the effort to get on the yahoo white list, since it benefits everyone involved for yahoo to filter as much spam out as possible.
Re:Who? (Score:1)
OK, but this isn't the problem the poster is talking about.. if yahoo's mail server stops accepting mail from a specific server, then the sender will get bounce messages.
So another example would be needed, as Yahoo isn't one of them.
Re:Who? (Score:1)
Re:Who? (Score:1)
Some do a reverse DNS lookup to make sure that the ip address the mail came from matches the domain of the sender.
Re:Who? (Score:1)
A friend in a band [slashdot.org] has a mailing list, that I've had to opt-into several times. I was a little annoyed at the ISP at first, but on reflection I wish more ISP were equally confrontational with their bulk senders.
Today, more than 50% of the e-mail I receive is SPAM. In the last 7 years, it's gone WAY past merely being annoying.
Make /var/log/mail public (sort of) (Score:1, Interesting)
Upon opt-in, issue each user a user identity (some random alphanumeric widget). Have a web page on your site that allows a member to enter their identity, and then a little CGI program parses
This is going to take a LOT of user education, but it's going to solve problems slowly over time. The emails that get dropped, if the user notices, will at least give your level one support something to go by. "Yes, our logs show that our mail server has delivered the newsletter to you on these days.... You didn't get it? Could you contact your ISP, and ask if they are filtering inbound email? Here, we'll email you the logs to pass along to your ISP, or you can get it from the web site."
To be polite, you could make the mail logs even more public, allowing the ISP to look up things, but you'd have to "sed" out email addresses, or at least obfuscate them (like everything left of the @ gets replaced by X's).
At the very least, it moves the technical problem from something vague behind the scenes to something more easily described, and seen, and comprehended, by the user. And it allows you to point the blame finger at the guilty party.
Finally, during the sign-up page, and on the troubleshooting pages you give to users, mention that if the newsletter doesn't arrive, a likely cause is their ISP. Give a top 10 list, based upon the problem frequency reports. (User changed email address, local mail filtering, ISP mail filtering, network outage....)
My worries (Score:2, Interesting)
What we've been doing is verifying our email lists (this goes a long way to avoiding getting flagged as a bad guy) and sending messages out one per connection. It's fabiously inefficient and it takes 4 hours to send out 12,000 emails (our biggest customer) but we've only managed to tick off about 3-4 other ISPs.
There's two things that I see as being issues that we're going to have to deal with soon in a real way:
1) Little Napolean wannabe sysadmins at other small ISPs that belive anything sent to more than one recipient is spam. These guys really irk me. Its one thing if their customer complains about mail from our domain and they evaluate the situation and block it but it's another for them to see a message destined for more than one mailbox on their domain and arbitrarly decide to reject all mail from our mail server (not just the domain that sent it mind you; ALL the domains we host.) Heart's in the right place but they left the lens cap on thier mind. I've tried talking with them but that just seems to iritate them more.
2) Big email hosting companies (Yahoo, AOL, MSN, Hotmail) looking to make yet another buck. Take a peak at these headers on a bulk email I got from Yahoo:
X-YahooFilteredBulk: 209.164.21.221
And this page from the Yahoo help desk:
http://help.yahoo.com/help/us/mail/spam/spam-17
Now don't get me wrong, I love (well, like) the bulk mail folder on my Yahoo account. I'm just waiting for these companies to decide to offer "Prefered Sender" subscriptions that will garante delivery to thier user's Inbox or maybe Prefered Partners Inbox or something. What are we (small ISP's) going to do then? We're not going to buy a subscription from every Yahoo/MSN/AOL out there and we can't serve our customers well if all thier lists get piped to
SpamAssassin! (Score:1)
SpamAssassin [spamassassin.org]
I'm not involved with this group, but from what I hear of other ISPs implementing this, it works well. It allows you to set headers based on it's own message rating system, sends checksums of messages that it thinks are spam to a clearing house (DCC), and uses checksums that match 'mass' email that have been rated as spam to mark messages that have been sent to a lot of people. This lets the user filter the garbage to a folder in their MUA if they want. It can also delete them server side.
Someone that uses this please correct me if I'm wrong.
Re: SpamAssassin! (Score:3, Informative)
The sysadmin running the mail server can have it do other things, like put likely spam into a different spam mail account that the user can check periodically.
Sounds dumb (Score:4, Informative)
Wouldn't this have a horrendously high false positive ratio for things like mailing lists?
Anyway, tell them to use SpamAssassin - it kicks ass. And I'm not biased, honest
Re:Sounds dumb (Score:2)
Re:Yahoo! is confused (Score:1)
Email is broken. (Score:2)
Email is never going to get fixed. The fundamental concept is flawed. You can't allow arbitrary messages from arbitrary anonymous sources without getting spam. Probably well over 99% of solicited mail is non-anonymous anyway, so the solution is simple, in theory.
Until anonymous email is deprecated the spam problem will not be solved, plain and simple.
it can be fixed ... Re:Email is broken. (Score:2)
Re:it can be fixed ... Re:Email is broken. (Score:1)
"just" require all SMTP traffic to use TLS, and have them all under one CA, so everone can test the authentication of the sender ..
Well, yeah, but if you're going to do all that why not throw out all the whole protocol altogether, or just require all messages to be PGP encrypted.
Spam is easy to solve in theory, but next to impossible in reality. Because we're stuck allowing backward compatibility, the spammers can always just pretend to be using the old broken protocol.
How about this? (Score:2)
I know it might border on heresy, but why not have the ISP actively manage the mailing lists? Here's an example:
Suppose I publish Gland Nut Weekly, and I use fatboys.net as my ISP. I register myself with the ISP, giving them the name of my mailing list, and the names/email addresses of the allowed publishers. When I have an issue ready to publish, I send it to fatboys.net, who then sends it to the current subscribers on the list.
Other ISPs can 'trust' that the email sent by fatboys.net isn't spam, since fatboys handles the mailing list, fatboys.net can be sure they're not a source of spam (and look like one of the good guys) since they're handling the mailing list, and the publisher benefits from having the ISP send the actual mail at high speed and without having to employ tricks to get around outbound spam filters. Whaddya think?
Re:How about this? (Score:1)
The whole story please... (Score:1)
Can you qualify this please? How many is "Many"? Two? Four? A hundred?
Worse still is that there is typically no error message returned to us - the emails simply get dropped
If this is true, then their mail servers are misconfigured, or your return address is wrong.
Are you sure you're not screwing up? Can you post your mail server logs showing that delivery has taken place?
If you're not getting bounces, then the ISP's are really accepting your email - which pretty much defeats the anti-spam logic (the whole point of anti-spam is to prevent mail transfer - which according to you, they're not doing.)
I'd guess that it's a problem with your equipment, or your mailing list software. Either your return address is wrong, or your mail server is dropping the mail instead of delivering it.
Sometimes, ISPs will add us to their "white" lists
OK, so you've contacted multiple ISPs, who all have their mail servers misconfigured in the same way, and you're convinced there are still more out there..
I think maybe the problem is at your end.
ISP? Give up. Are they your employees? Keep trying (Score:1)
That said, I also think that all emails should be PGP signed, and all emails that fail in THAT regard should be summarily filtered... (of course the process to get there could be as gradual as having the email client flag unsigned messages as "suspicious", yadayadayada... so as not to shock the masses with a sudden change... blah blah)
Re:ISP? Give up. Are they your employees? Keep try (Score:1)
Becuase, of course, spammers are too stupid to download PGP and make a key.
Why on earth does this pop up in any anti-spam discussion? PGP signing simply means the sender can prove it was from him. It doesn't mean you know who the sender is.
If you want to set up some sort of whitelist, it makes just as much sense, and takes much less space, to say 'I will accept email from blah@mail.dom, and only if it arrives via mail.dom or dialup.dom.'.
If you want to do something useful with PGP, you could make something where you auto-whitelist anyone who has a key signed by someone you trust. That's about the only way PGP can help fight spam.
Re:ISP? Give up. Are they your employees? Keep try (Score:1)
And you are exactly right about keeping a list of valid PGP signatures, since the one thing I don't want a spammer (or con artist) to be able to do is fake being someone I know and trust.
Re:ISP? Give up. Are they your employees? Keep try (Score:1)
Plus, that's easily solved, if they actually start doing that, by saying 'I will only accept mail from whoever@server.dom, and the only machine that can send me that mail is server.dom.'. If someone has a weird situation where email doesn't arrive from the machine server.dom, you simply give them an exception.
PGP signing is so that you can prove later they sent it, not so you 'know who it's from', it's trivially easy to figure out if an email is from someone you know just by looking at the headers. If a friend always PGP signs his email, sure, accept that as proof it's from him. But don't make everyone start signing things, being from the right server with the right email address is proof enough it's not a spammer.
Re:ISP? Give up. Are they your employees? Keep try (Score:1)
ermm.. not really. maybe i'm just paranoid, but AFAIK the best headers can do for you (without disruptively contacting system administrators to discover mac addresses) is narrow down the subnet that the message came from. Most ISP's that i am aware of have open smtp relays within their subnets.. i.e. anyone within the subnet could pretend to be anyone else within the subnet and nobody could know the difference.
Granted that most Outlook-using users and spammers wouldn't have a clue how to do this, but anyone who can understand the command-line syntax for sendmail can do pretty much whatever they please.
Re:ISP? Give up. Are they your employees? Keep try (Score:1)
This is so far from reality I don't know where to start. Spammers run software that looks for things like blah@example.com. This is the entire extent of their 'finding email addresses'. They not only don't do any of these complicated things you're talking about to figure how to get in past one address, they don't even filter out obviously wrong addresses. Spammers sometimes send to Usenet Message-IDs, which only look like email address if you're just globbing *@*.???, and don't bother to look and see it's jf3224-usieof.disuwod@example.com.
If it takes a spammer an hour to send a message to a person, they've lost and we've won. Hell, if it takes a spammer one minute to send a message to someone, we've won. Spammers are sending out something like a million messages each time, and each run needs to be done in a few hours.
Re:ISP? Give up. Are they your employees? Keep try (Score:1)
If it takes a spammer an hour to send a message to a person, they've lost and we've won. Hell, if it takes a spammer one minute to send a message to someone, we've won. Spammers are sending out something like a million messages each time, and each run needs to be done in a few hours.
I agree with you at least this much.
So, you're assuming spammers are sniffing your email and finding out not only the names and address of your friends, but what headers they send with their message, and searching until they find an open relay within the right subnet so they can send using the same SMTP server as your friend?
ArggggghhhhH!!! NO! I said already (several times) that I come on the side of not particularly caring if I get spam. Bandwidth isn't even an issue for me since newer clients (like the newest kmail) can filter based on subject and sender while the email is _still on the server_.
All I want from my email is to know (beyond a reasonable doubt) that the person who sent it to me is the person I think it is. I also want to know (beyond a reasonable doubt) that it would be impossible for another person to forge an email from me to someone else without that email being red-flagged as suspicious.
However, if the above properties were true of email, it would be very hard for spammers who send gazillions of anonymous emails to get any attention, since those emails could be sent into an "anonymous" pile which rarely gets looked at (since it's full of spam).
The other emails are PGP verified in a way that should not reveal the email address doing the verifying, eg. the final server could verify the authenticity of each incoming email, valid or invalid, and modify the headers to reflect the authenticity of lack thereof.
Once a client receives an email, one of the things it would be able to do is look at the headers to see if the email is valid or invalid, and react accordingly by sorting or doing whatever user-defined action it is supposed to do. Older clients can hopefully just ignore the strange new header. If a person reading an email is particularly interested in knowing if an email is valid or invalid (i.e. if they think the server might have made a mistake, or they don't trust the server), the person can click on a button that checks the authenticity of that message manually. The other thing that the person can do now (which they couldn't do before because of anonymous emails) is COMPLAIN about the unsolicited email, and have a solid line of accountability leading straight back to the spammer's server.
At the very least, the problem for spammers would have moved from finding open smtp relays to finding open httpd servers (much harder to find)....
Actually, why haven't ISPs adopted some form of us (Score:2)
I suggested something like this a while ago. Server side filters accessable by ordinary users. People here said they have those, but misunderstand. Most server side mail filters apply to ALL accounts and are not accessable by users who have pop accounts. In fact I have not heard of an ISP implementing such an idea and I claim this as prior art for such an idea so don't even think of patenting it I'll sue.
It's simple, a users logs into their isp with a web based app that allows them to say filter out this that and blah. I'd use mail headers, and filter out korean character sets as that is where most of my spam lately comes from. Funny I can't even read it but the charset says korean.
I am leaning alot about smtp / pop and basically the only requirements are HELO, MAIL FROM, RCPT TO, DATA, QUIT, USER, PASS, etc. The protocols themselves are too stupid for most else. Filters on the server could also interfear with privacy. In order for them to filter mail they would have to have a mail scanning program. If they log this data then it becomes an provacy issue.
The real solution is better mail filters in the pop mail cleints. For a delete filter it may be better if the pop client were to call TOP and get the message header and then delete the message appropriately. I am working on a java implementation of this. My POP3 bean can do this, I just need to scan the headers.
Re:Actually, why haven't ISPs adopted some form of (Score:2)
it is possible to execute the TOP command and download the headers of mail and from the mail headers have it delete mail based on that. TOP 1 0, gives me just the mail headers. If I have 20 spam messages and I just get the headers of them I can delete all the spam and not download the whole message. I do this through my web based application that I have where I display the inbox I only get the headers. Maybe the soultion is to leave the mail on the server and only get the headers in the mail app and then select which messages I want to download after that. I could also set up filters based on these headers so that I never see the messages in my inbox that have lets say a character set that is in another language other then my own preference.
headers are usually less than 1k, but html spam is usually several k. This would cut down on my download time.
Re:Actually, why haven't ISPs adopted some form of (Score:2)
From 'spamassins web site'
The 'user agent' it the users mail program. This means that the users is not filtering out the data on the server. The server is only 'tagging mail'. The user still has to download the whole mail. Obviously your to stupid to understand a thing I am talking about. I am talking about a filter on the mail server that I set up that delete the mail from my inbox and I never ever see it. So in my case I would create a filter that says 'delete mail where charset like "korean"', then all mail that is coming from korea is deleted form the web server when it arrives at the pop mail account on the mail server.
My ISP uses the spaminator which reduces my spam by over 50%, but it is still not a filter that I set up for my account on their servers.
Its obvious from your post that it doesn't require brains to post on slashdot.
First, stop sending to peole don't want it. (Score:1)
However, you luckily aren't on any blackhole lists. Yet.
And it's a problem with your mailer. All anti-spam software returns errors to your mailer when you connect, or bounces the email. It wouldn't drop them on the floor, that's not discouraging you at all, you'll still keep sucking up their bandwidth, as you can't possibly know they're being dropped.
Ergo, your mailer does not understand the 5xx reply they are sending. You need to report it as a bug.
Re:First, stop sending to peole don't want it. (Score:2)
Nope. Not all. Perhaps it is supposed to, but not all does. Especially at an ISP. I've sent mail from one of my email accounts (that I pay for) to another (that I also pay for), and the second location just drops them off to the bit-bucket.
Remember, if someone falsifies mail origins, kicking back won't help as much. Or the filtering might kick in a little later in the ISP's server chaining. Or the ISP might feel that would be like supporting the VRFY command, which most do not nowadays just for spamming reasons.
Re:First, stop sending to peole don't want it. (Score:1)
I dunno, though, if it's truely some sort of message counter, it might accept them all and retroactively delete them if there are more than X. I can't comprehend someone actually using something that, though, I was really working on the assumption that he's in a private blacklist or something. I thought at first he'd ended up in some public one and didn't know it, but I can't find him anywhere. But he talks about this happening on multiple providers.
Message counting doesn't make any sense, and I've never heard of anyone doing it, at least not for a domain. It's simply too much work to keep track of mailing lists. Maybe he's ending up in Vipul's Razor or something. (Which is certainly possible, as he's not using any sort of confirmed opt-in.)
Of course, as he's not using confirmed opt-in, I don't really want to help him, beyond 'use confirmed opt-in'.
Re:First, stop sending to peole don't want it. (Score:2)
Ahh. But a large issue is that if a spammer issues a bunch of mail into your server, and some of them are accepted and some of them return errors... then suddenly the spammer has a way to check if addresses are live or not, and has a replacement for the VRFY command.
I'm not saying that this ISP behavior has good reasons, just that it has some reasons. And for some ISPs, that's reason enough. Really sucks for legitimate users, though.
I use A-S-K (Score:1)
http://www.paganini.net/ask
or
http://sourcef