Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

What E-Mail Validation Tools Do You Use?

Posted by Cliff on Wed Nov 01, 2006 06:01 AM
from the return-to-sender dept.
Spam
morcego asks: "As we are all too much aware, spam is an increasing problem. Each of us has our own set of tools and methods to try and reduce the amount of spam we receive, each with different pros and cons. Also, on a more broad front, we have options like SPF (+ SRS), Microsoft's own Caller-ID, and Yahoo's DomainKeys that we can use. These days, it is incredibly easy to implement any (or all of these), using publicly available frameworks and libraries (libspf2, and milter, to name a few). I have been using SPF for quite some time now with some measurable results, although nothing earth shattering. Which of these are you using, if any? Why, or why not? Do you think any of them really contribute anything to fight spam?"
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

What E-Mail Validation Tools Do You Use? 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • I don't use anything other than dspam. It filters 99% of my spam for me. What more could I want?
  • The problem is that they can be spoofed, although not quite easily. That's because they're having folks self-setup the various systems.

    Me, I would rather say "If your domain isn't in the same netblock as the ISP it represents, score heavily against."
    • I don't see spoofing as the problem. I see critical mass as the problem. Unless nearly ALL ISP's and email systems adopt a single "standard", the mechanism is useless. We don't have critical mass. I'm seeing less than 1% adoption rate for any of these systems.

      Furthermore, these systems are not designed as anti-spam systems. Phishing and JoeJobs they may help with. Spam not at all. Since they don't help fight spam, there is no incentive to adopt them.
      • 1) They eliminate sender spoofing in emails
        2) Without sender spoofing, you can see what domain an email actually came from
        3) Ban the bad domains in your emails rules.

        wow
        • 1) Without critical mass, it doesn't.
          2) No shit.
          3) Ban all you want. Domains are cheap so spammers will create more...

          End result - no change at all in spam volumes. If all the big ISP's got together and said that in January 1, 2008, they would no longer accept mail from anyone without an SPF record / SenderID, you MAY get 70% compliance. But I doubt it. In order to be truely effective, you need 90%+ compliance. Even at 100% compliance, you won't fix #3.
          • All I'm saying is that this is the only way it'll happen with current email tech. Any if you have spam filters that flag 100 spams coming from a domain, ban the domain automagically. I would think that would work *really* well, if like you said we ever got critical mass of SPF capable ISPs and began requiring it. The only reason they havn't, in my opinion, is that they make more money off the Spam themselves by selling anti-spam crap to their customers.
      • I'm just going out on a limb here. Let's say I send you an email. I come from hksilver.net. If you resolve hksilver.net, it would return (currently) 208.231.66.99. Now, right away you're going to come across a problem. I host my own domain, I have gotten my ISP to put a PTR record for my host. So reversing it will return mail.hksilver.net. If you were to go by netblock 208.231.66/24, and you're checking to make sure the email sources from that netblock, you'd be okay in my case about 75% of the time.

  • Mailvisa (Score:3, Informative)

    by RAMMS+EIN (578166) on Wednesday November 01 2006, @06:43AM (#16671011) Homepage Journal
    I wrote my own Bayesian filter, Mailvisa [inglorion.net], to gain a better understanding of how Bayesian filtering works, and to be able to tweak the parameters. When I last measured it, it caught 93% of spam. Of all the filters I tried at the time (I think it was all filters in Debian sarge), only Bogofilter [sourceforge.net] scored better. This applies to both the amount of spam caught and the filtering speed. The closest thing to false positives I've gotten over the years were a few advertisement mails from my domain registrar.

    I have only two problems with it: 1. I have to train it regularly, and 2. nowadays, lots of mail slips through, because it contains words related to programming languages.
  • Greylisting and DSPAM work for me. The odd spam still gets through, though the majority of those can be rejected with various postfix settings.

  • I use GMail :) (Score:4, Interesting)

    by brunes69 (86786) <<gro.daetsriek> <ta> <todhsals>> on Wednesday November 01 2006, @06:48AM (#16671039) Homepage
    After trying to tune SpamAssassin to work well for months, and being unimpressed by the hit/miss rate, I tok to forwarding all of my incoming email to GMail. I then forward all my email from GMail that is not spam back to my other account :0

    I find this way I get 99.95% accuracy - things that GMail misses as spam, my local SpamAssassin catches. As a side bonus I have GMail's awesome interface to read my mail when on the road (much better than the Squirrel Mail I was using, and still better than RoundCube).

    This brings up another point - I don't know why Google doesn't add IMAP connectivity to GMail, soyou could use it's interface to read email from other hosts. I don't see why their ad technology would not work with this scheme.

    • ``This brings up another point - I don't know why Google doesn't add IMAP connectivity to GMail, soyou could use it's interface to read email from other hosts. I don't see why their ad technology would not work with this scheme.''

      In fact, if they can forward your mail to another account (which they do) and they can offer POP3 (I think they do), they can offer IMAP, too.
    • The problem is it becomes painful to view your mail in the other account. Unless you have an automatic filter somewhere to strip the gmail headers. Every mail would appear to have come from your gmail account. The sender of email is a useful thing for me to keep track of my mail.
      • I have gmail auto-forward to my work account due to stupid webmail blocking policies. If I click "reply" in Outlook, the "To" address is not my gmail account; it is the person who sent the mail originally.

        Oh, and the "From" field in my Outlook inbox shows the correct sender.
      • Not true, GMail preserves the original headers when it forwards.
    • I'm no expert, but doesn't this mean you'd have to check in two places for false positives?
  • I get about 99% success with Spamassassin. (I do train it on its errors, about every couple of weeks.) The most common leakage I was getting was bounces from domains when the spammer spoofed my domain name; I finally put an SPF record in place, and those seem to have stopped.

    One thing I wish it would allow would be to train it on all rules, not just those that the Bayesian filters use. Some of the rules give me a lot of false positives, but they'd be fine for others: so why do we have to manually change
  • While not necessary e-mail validation tools, greylisting and SBL+XBL blocking lists by Spamhaus have eradicated nearly all spam I used to get through all of the other filters.

    Greylisting alone helped to lower e-mail traffic drastically and blocking lists take care of known spamming hosts. I'd recommend using both to anyone running a e-mail server.
    • Don't expect Greylisting to reduce spam for long. Spamware is evolving and will start taking greylisting into account shortly, much like image spam gets around bayesian analysis. It's a matter of time before spammers start snagging email configuration info (such as SMTP Auth info) from pwned machines and sending spam via normal ISP gateways. Even rate limiting won't help as the number of pwned machines is massive, and growing every day.

      BTW, even OCRing (which is very expensive computationally) of image spam
      • ...some spam does get through, but very very little (on the order of 5 per day. 5!).

        Er, what's the context for that "5"? If your company only gets 100 emails a day that 5 is actually pretty lousy. (Now if they get 100K a day then it's great!)

  • spf,CallerID,DKIM validates sender only not Spam ! (Score:3, Interesting)

    by johnjones (14274) on Wednesday November 01 2006, @07:27AM (#16671257) Homepage Journal
    all that SPF CallerID and DKIM does is validate the sender !

    this cuts out about 70% of (stupid) spammers
    you also need to blacklist people who send you spam (and you can be confident that you get them because of the above technologies)
    if you Ever want to send lots of mail to hotmail users you need to have callerID setup yahoo and gmail both trust you more if you have domainKeys
    so things are moving on and there is no reasson why people should not have at least one of SPF CallerID or DKIM setup on their domain !

    you will note that people here also use filtering but the question is does the filtering feedback to the blacklists ?

    regards

    John Jones

    p.s. I work in the mail vendor world...
     
  • SpamBayes. After enough training it is spookily accurate at getting spam. I used to run SpamAssassin as a POP3 proxy and then filter the rest with SpamBayes, but recently (past year or so) SpamBayes has been enough.

    This *might* be due to ISPs doing a better job of bulk filtering out the obvious junk before we even see it. Some of the domains I have that are on other than my main ISP do seem to end up with more spam, but after filtering via SpamBayes I see very little...

  • pf OS fingerprinting (Score:3, Informative)

    by jnieuwen (524859) on Wednesday November 01 2006, @07:31AM (#16671289) Homepage
    I use the OS fingerprinting options from pf to block windows machines from delivering mail on the primary mx. This saves approximately between 300 and 1600 spams a day. Beside that, rejecting mail from hosts without an A record, blacklisting all hosts sending mail to spamtraps with spamikaze [spamikaze.org], rejecting hosts which falsely claim to be a host in my domain and filtering with bogofilter.
    • blacklisting all hosts sending mail to spamtraps

      So you blacklist all mail from yahoo, hotmail, gmail, msn, aol, verizon, earthlink, etc.? Because all of those servers send to spamtraps all the time.
        • That makes very little sense. The big ISP's don't have one email server. They have hundreds. What will happen is that you will eventually blacklist all of them, and when a user gets a bounce, he can whitelist ONE of the servers, send his message again and get another bounce because he hit yet another bkacklisted server.

          You are better off maintaining a per SENDER whitelist rather than per SERVER to be effective in this scenario (which is what we do for "evil" domains like yahoo and such that are heavily used
  • This is the list of most of the stuff we run at the border:

    Exim + greylisting +c lamav + Spamassassin.

    Here are the plugins to spamassassin and custom rulesets:

    Plugins:
    ---------
    Razor2
    SpamCop
    AWL
    MIMEHeader
    ReplaceTags

    Custom Rulesets
    ----------------
    We use a selection of the SARE rulesets
    70_sare_adult.cf
    70_sare_bayes_poison_nxm.cf
    99_FVGT_Tripwire.cf
    bogus-virus-warnings.cf

    This was stopping most of our spam...however we were still getting a lot of spam that contained images with the spammy message. So about 2 weeks
    • add the sare_stock and the FVGT rules, this'll stop the stock image with the huge overhead of fuzzyOCR.

      also have a look at the other SARE and jennifer rules - I find these very useful.
      • I've already seen a new technique to defeat the OCR software. Yesterday I got my first email with a spam message that contained a single image for each letter of the message. Of course FuzzyOCR didn't hit on this. Not sure how we'll get around that one.
  • SPF (and related technologies) are not designed to cut down on spam. They are designed to prevent Joe jobs [wikipedia.org] and address forgery. (It just so happens that most Joe Jobs are spam).
    • SPF records can be useful to identify legitimate e-mail servers from selected domains.

    • SPF (and related technologies) are not designed to cut down on spam. They are designed to prevent Joe jobs and address forgery.

      I just went through this with a security company for a Visa audit, so let me expand on this. They seemed to think that checking the Mail From: for a local user, when sender wasn't authenticated (I would assume - we never actually got that far), was a valid way of checking for forgeries. There are multiple problems with this.

      • Their testing was flat out wrong to begin with. Th
  • The combination of 8 DNS blacklists, Amavis and Spamassassin works very well.
    I used to get more than 300 spam mails per day (intercepted by Spamassassin), due to the use of DNS blacklists I now only receive about 15 spam mails per day wich are intercepted by Spamassassin.
    Only about 3 spam e-mails per day actually make it into my mailbox, with zero false positives.

    The good thing about DNS blacklists is that the spam e-mails are actually rejected in the mail protocol, therefore it will hit spammers directly a
    • The blacklists also reject dynamic ip addresses, which are all virus infected home computers.

      *All*? I run a mail, gaming, and web server off of a dynamic IP. Forwards out through a smarthost, so blacklisting isn't a problem, but it isn't infected with viruses nor am I using it for illegit purposes (ok, well it probably does violate my ISP's TOS, but fuck'em).

      -b.

  • I'm the entire IT Dept at my work and I do not have the time to manage our own email server, let alone worry about keeping it secure. Most of our business comes in via email and most of those are crafted to look exactly like spam with huge lists of names in the TO: or CC: boxes and no subject line.
    My problem was finding a way to filter spam without filtering even a single legit email. Lost email means a lot of lost revenue. The only solution I found in a year of searching was mxlogic.com. We still get spam,
  • And that's been keeping the ones that get through down to two or three a week. Not enough for me to turn on hard SPF checking or demanding that email to me be encrypted with my personal PGP key. Configuring all that stuff certainly is a pain though -- it'd be nice if they could get it down to drop in components for the most common configurations.
  • The postfix server uses RBLs to drop about 25,000 messages per day. If postfix accepts it, it gets handed off to a different server that does SURBL checks. (That is done by a commercial product called GWAVA [gwava.com]). The SURBLs catch about another 2,000 messages per day.

    I have published my SPF data - so at least other people have the option of identifying whether stuff that claims to have come from my domain is legitimate or not. But our mailers are not yet doing SPF lookups. When we have a little time, we will pr

    • called GWAVA

      You're running Groupwise? GWAVA is overrated and is mainly useful for integrating spam filtering into Groupwise's Internet Agent. Nothing that SpamAssassing + ClamAV + ProxSMTPd won't do for you. And that combination is available as part of a package for an IPCop firewall box called CopFilter. The only downside is that CopFilter isn't as configurable as it should be via the Web interface. But for a free product, it's pretty darn good.

      -b.

      • That's cool. If we got into a monetary crunch, we would probably implement what you mention here. One of the nice things about GWAVA is that it we have it configured to send an HTML message to users daily, where they can pull a message out of the bit-bucket (so-to-speak). That is to say, they get a digest message of what was blocked, and if something was improperly blocked, they can have it sent to them anyway. Version 4 (due out any minute now) will take this to the next level, where users can manager thei
        • One of the nice things about GWAVA is that it we have it configured to send an HTML message to users daily, where they can pull a message out of the bit-bucket (so-to-speak). That is to say, they get a digest message of what was blocked, and if something was improperly blocked, they can have it sent to them anyway.

          Copfilter has a digest option too. We're not using it ATM, since I have it set up to block only the most egregious examples of SPAM i.e. those with scores of 25 or above. The rest simply gets

  • I use a (usually) sophisticated biological neural network consisting of a multi-billion plus nodes with some primitive pre-determined wiring structures serving as a foundataion. Oh yes, and as preliminary step, I use dual-stage filters: spamassassin followed by crm114. Spamassassin seems to be fairly well behaved by not giving too many false positve spam indications, and CRM114 picks through the remainder false negatives to my satisfaction. I still end up picking through the spam folders, but its bulk and n
  • They've done so for the past few years, and it seems to work *very* well.

    See their web site here [postini.com]...

    • > They've done so for the past few years, and it seems to work *very* well.

      My previous ISP imposed Postini on me with no notice (they sent me an email bragging about it three days after they started using it). It passed 50% of the spam and stopped 20% of the ham. I turned it off.
      • Interesting. My ISP introduced it as an opt-in service (just like they introduced SpamAssassin and various other tools to the user base), and while it did require some fine tuning, I've had very few problems with it (I get a handful of Spams a day which it doesn't catch, and I see one or two false positives a month).

        I don't blame you for dropping it given how it was introduced at that ISP, but I think you also lost a chance to use a fairly effective anti-spam tool.
  • In other words, it's crochety as hell. I have all the "speak the RFC's exactly or thy shall not pass" options turned on. I publish a SPF record, for what good it will do. I also 5xx reject anything from overseas.

    Even though this is my own personal mail server, I haven't had too many false positives as far as rejects go... certianly nothing that a tweak here or there in the allow/deny hosts file wouldn't take care of.

    All in all, I've recieved less than a dozen pieces of spam in the last year and a half. Not

  • Greylisting (Score:3, Informative)

    by eric76 (679787) on Wednesday November 01 2006, @12:45PM (#16675459)

    I use spamd on OpenBSD to do greylisting. That cuts an enormous amount of spam out.

    For those who aren't familiar with greylisting, when an smtp server attempts to deliver an e-mail the from address, to address, and IP address of the sender are put in a database and the mail is refused with a non-permanent error code.

    Assuming the smtp server sending the e-mail follows the RFC, it will try again later. When it tries again after at least 20 minutes from the original attempt, it accepts the e-mail and adds the IP address of the source to a whitelist. For the next 30 days, any e-mails from it are white-listed. After that, the server is verified again.

    I also keep a seperate white-list for non-RFC compliant servers and for frequent senders. Some servers only try one to three times and quit. Another problem is e-mail from some large e-mail farms may make each attempt to deliver the e-mail from a different server with different IP addresses, so I'll add their e-mail addresses to the white-lists as well.

    One method I use for adding IP addresses of selected senders that send a lot of legitimate e-mail to the whitelist is to look up their SPF records and use that to identify the usual e-mail servers for the domain.

    A few ISPs appear to put their entire address space in the SPF record. For example, panix.com's SPF record is

    panix.com text = "v=spf1 ip4:166.84.0.0/16 ip4:198.7.7.0/24 ?all

    Needless to say they don't get whitelisted since I only want to whitelist e-mail servers, not their users spam-zombie computers.

    In other words, I use the SPF records to identify legitimate e-mail servers from selected domains only.

  • I used to "roll my own" with SpamAssassin and MimeDefang. Then I started using CanIt [roaringpenguin.com] at work (I liked them initially because the author is the author of MimeDefang). They have a free version that works well for me at home now. We have been using it for about 4 years at works and it does a great job incorporating grey listing, SA, MimeDefang, ClamAV, etc. into an easy to install and maintain system with a nice web interface and a database backend. It can scale well when we need it to and the support is great


    • We were using MimeDefang + SA for a while, but it wasnt enough. Second the vote for Canit... just (as in Wednesday) rolled out Canit/PRO to serve mailboxes for 5000 full-time employees. Works well, cost is very reasonable. It has the benefit of the centralized solution for reduced maintenance, but we can use the web interface to customize mail flows for people with particular needs.

  • I pay poor children in [???] $0.01 / hour to filter my mail for me. It's cheaper then buying SPAM filtering software.
  • We use ASSP at work (a government entity) and it is effective enough that when we DO have a spam slip through, users usally call to complain about it. It happens rarely enough that they forget to forward it to spam@<ourdomain.org>.

    I also use it at home and have nearly the same effectiveness.

    As far as various technologies, I don't believe any solution which relies sole upon one or two technologies will be that effective. ASSP seems to be the best so far at combining SPF/Greylisting/bayesian/various oth
    • Me too. If I don't recognize the sender OR understand within one second what the subject header means, the mail goes in the bin. I don't get much mail from unknown persons at work, but if I do these persons better make sure their subject header is to the point and understandable otherwise it's their bad luck.

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.