Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Spam Technology

Computationally Cheap Spam Filtering? 85

Posted by Cliff from the cheap-and-inexpensive-crap-detection dept.
Roadmaster asks: "Usually, the most effective spam filtering techniques are somewhat resource intensive. Heuristic checkers like Spamassassin, or bayesian filters like spamprobe are processor and storage hungry. This is fine for small setups; I've been using spamprobe to filter spam for 3 users with great results. I'm now however faced with a big challenge: a mail server that will eventually be handling mail for over 50,000 users and needs to have some sort of anti-spam measures. What are some good and computationally cheap spam prevention measures?"

"Ideally, I'd prefer something that does reject the message if it's spam (SMTP result code 550 or something like that), unlike current Spamassassin or spamprobe setups that accept the message and only later decide whether it's spam. Solutions like MAPS RBL, ORBS are acceptable altough commentary on their accuracy would be welcome. Other possibilities I've thought of include checksumming (Vipul's razor or DCC) and simple header checks that could be implemented for instance in a sendmail milter.

Are several quick checks (DCC + RBL) accurate enough and still cheaper than one slow check (Spamassassin, bayesian filtering)? does stacking of similar techniques improve accuracy significantly? (DCC + Razor, RBL + ORBS). How can the good but expensive techniques be made cheaper? (Spamassassin's spamproxyd, hashed wordlists for bayesian filters, and so on). Discussion on all these aspects would yield some interesting conclusions on quick and efficient spam filtering."
This discussion has been archived. No new comments can be posted.

Computationally Cheap Spam Filtering? More

Computationally Cheap Spam Filtering?

Comments Filter:

  • FP? (Score:2, Insightful)

    by GrendelT ( 252901 )
    what kind of platform are you running from? if you'll have that many clients to support, you might consider having a dedicated spam-filter. that way you dont have to worry about resource-hogging filters

  • Solution (Score:4, Interesting)

    by m0rph3us0 ( 549631 ) on Thursday May 08, 2003 @04:16PM (#5913202)
    Personally, I would recommend going with spamassasin, RBL and stuff gets too much non-spam and causes user problems. I would recommend that next to your mail server you setup a load balancer + DHCP server. Then modify a copy Knoppix to run as a spamassasin server and use the mysql for configuration of spamassasin, then buy several cheap consumer level PCs and plug them in for cheaply scale your processing power. Then run spamassasin for 50,000 users.

  • Hardware Virus Checker (Score:5, Informative)

    by Kalak ( 260968 ) on Thursday May 08, 2003 @04:17PM (#5913212) Homepage Journal
    At our uiversity, Virginia Tech, the hardware e-mail virus scanners (Mirapoint Messaging Server )also do Spam Assassin now, it puts info in headers (sample below). Filter for "X-Junkmail: UCE" and you've got a spam filter (though I run a more aggressive SA on my workstation, since I can customize it there).

    Return-Path:
    Received: from vt.edu (gkar.cc.vt.edu [198.82.161.196]) by xxxx.xxxx.vt.edu (8.12.8/linuxconf) with ESMTP id h47JISRm004277 for ; Wed, 7 May 2003 15:18:28 -0400
    Received: from steiner.cc.vt.edu ([10.1.1.14]) by gkar.cc.vt.edu (Sun Internet Mail Server sims.3.5.2001.05.04.11.50.p10) with ESMTP id for noone@xxxx.xxxx.vt.edu; Wed, 7 May 2003 15:18:31 -0400 (EDT)
    Received: from aol.com (host217-40-92-155.in-addr.btopenworld.com [217.40.92.155]) by steiner.cc.vt.edu (Mirapoint Messaging Server MOS 3.3.2-CR) with SMTP id BIE36579; Wed, 07 May 2003 15:18:17 -0400 (EDT)
    Date: Thu, 08 May 2003 03:13:26 -0800
    From: Kate Welsh
    Subject: [SPAM] Remember me?
    To: spam@vt.edu
    Message-id:
    MIME-version: 1.0
    X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
    Importance: Normal
    X-Junkmail: UCE(58)
    X-Priority: 3
    X-Spam-Status: Yes, hits=13.0 required=5.0 tests=ALL_NATURAL,BASE64_ENC_TEXT,BIG_FONT,CLICK_B ELOW, CLICK_HERE_LINK,DATE_IN_FUTURE_12_24,HGH, HTML_FONT_COLOR_CYAN,HTML_FONT_COLOR_GRAY, HTML_FONT_COLOR_NAME,HTML_FONT_COLOR_RED, HTML_FONT_COLOR_UNSAFE,HTML_FONT_COLOR_YELLOW,NO_Q S_ASKED, RCVD_IN_DSBL,REMOVE_PAGE,SPAM_PHRASE_13_21,SPAM_RE DIRECTOR, SUSPICIOUS_RECIPS,USER_AGENT_OUTLOOK,VERY_SUSP_REC IPS version=2.44
    X-Spam-Flag: YES
    X-Spam-Level: *************
    X-Spam-Checker-Version: SpamAssassin 2.44 (1.115.2.24-2003-01-30-exp)
    X-Spam-Prev-Content-T ype: MULTIPART/MIXED; BOUNDARY="Boundary_(ID_d+Bzp/dF6h/2OkPD89OTbQ)"
    C ontent-Type: text/plain; CHARSET=US-ASCII
    X-Evolution-Source: imap://jackie@localhost/
    • I just looked around on Mirapoint's site and I can't find any reference to SpamAssassin. Where did you hear that's what they use? Or did I misunderstand the post?
      • Well if he's getting this line in his email headers:

        "X-Spam-Checker-Version: SpamAssassin 2.44 (1.115.2.24-2003-01-30-exp)"

        Then....
      • Our mail server admins let VT know about it during testing. It was recently purchased after a successful trial. I'm not sure the arrangement, but you can probably ask postmaster at vt dot edu and get a good referral on the setup.
        • That's kind of interesting because McAfee recently bought DeerSoft which was the only commercial provider of SpamAssassin I could find recently when our corporate powers-that-be mandated that while we would have a spam filter and that it would not be open-source.
          Think I could get an "official" statement from MiraPoint that it uses SpamAssassin? We'd rather not wait for McAfee to get their act together...
          Any contacts there I could call/email? (If you don't want to reply publicly I can be reached vi

  • Client side? (Score:2, Informative)

    by Drakon ( 414580 )
    how about filtering the mail client side?

    For outlook users, i recommend Spammunition [upserve.com] and I just use mozilla's spam filtering, which works great.

    • Client side: Eudora (Score:3, Informative)

      by User 956 ( 568564 )
      For outlook users, i recommend Spammunition [upserve.com] and I just use mozilla's spam filtering, which works great.

      Eudora users can use Spamnix [www.spamnix]. Works like a charm.
    • Query: Why is it that cool software for outlook never exists for outlook express?

      Do the authors get a special deal from M$ for not making their software work with express?
      • Because Outlook is based on MAPI (allowing a certain degree of plug-in behaviour) and has its own plug-in architecture which makes it very extensible, whereas Outlook Express is (was?) much more monolithic.

        Very different programs in all but name.

  • Some Ideas (Score:2, Interesting)

    by rmull ( 26174 )
    Caveat: I know very little about email in the real world. These are just my thoughts.

    What are your requirements? Do you have very limited hardware to work with? Do you need a particularly low latency for delivery? How many messages do you need to process per minute? (or per second)

    If it's possible, having a seperate spam filtering box might be a good idea. If that gets loaded down you could even make a cluster of them. I'm not sure that high-level spam filtering really takes as much cpu time as you'

  • Another cheap way.... (Score:3, Informative)

    by m0rph3us0 ( 549631 ) on Thursday May 08, 2003 @04:20PM (#5913254)
    Use TDMA http://tmda.net/.

  • A few comments... (Score:5, Informative)

    by aridhol ( 112307 ) <ka_lac@hotmail.com> on Thursday May 08, 2003 @04:21PM (#5913263) Homepage Journal
    Ideally, I'd prefer something that does reject the message if it's spam
    Careful. I'm assuming that these will be clients or co-workers or similar. BE CAREFUL. You do not want to drop messages. What happens if a client's email is lost because it looks like spam (refers to money, etc). Better to tag it, and let the user decide.

    Are several quick checks (DCC + RBL) accurate enough and still cheaper than one slow check (Spamassassin, bayesian filtering)? does stacking of similar techniques improve accuracy significantly? (DCC + Razor, RBL + ORBS).
    SpamAssassin is a stacking of checks. You can set up its config to skip those checks that you don't want to bother with. You may need to adjust scores if you do that, however.

    • Re:A few comments... (Score:3, Interesting)

      by Harik ( 4023 )

      Careful. I'm assuming that these will be clients or co-workers or similar. BE CAREFUL. You do not want to drop messages. What happens if a client's email is lost because it looks like spam (refers to money, etc). Better to tag it, and let the user decide.

      Um, don't even bother. Either filter and drop the spam, or just let everything through. Having someone go through all the marked spam messages is just as wasteful as going through the unmarked ones. If you're that afraid of dropping something, conside

      • Re:A few comments... (Score:5, Insightful)

        by aridhol ( 112307 ) <ka_lac@hotmail.com> on Thursday May 08, 2003 @05:28PM (#5913945) Homepage Journal
        If it's for a company email system, make sure that the higher-ups know exactly the implications of this system. And I don't mean just the CEO. Make sure that every department head knows. Go to a meeting with them first, to discuss it. Do not enforce your own policy. That's not your job.

        Make sure that people know that they can (and probably will) lose legitimate email. Make sure there's a way to bypass the filters. For example, hold the email until you can confirm the sender (reply to sender, and if your message bounces or isn't replied to in n days, delete). Let users setup their own configuration (scores, whitelists, etc), but be able to override some things (eg don't let them blacklist internal mail).

      • Re:A few comments... (Score:3, Interesting)

        by tdelaney ( 458893 )
        I would disagree strongly here.

        I use spambayes [slashdot.org] for my spam filtering.

        I get about 50 items classified as spam per day. These have a spam probability according to my spam and ham corpuses of > 90% (usually 100%).

        However, I also get 2-6 things classified as *possibly* spam each day. These are things which have a spam probability of > 15%.

        These get mixed up in among 200-400 other messages each day.

        Once I got things set up, I have *never* had anything classified definitely spam which wasn't.

        Most of t
    • Careful. I'm assuming that these will be clients or co-workers or similar. BE CAREFUL. You do not want to drop messages. What happens if a client's email is lost because it looks like spam (refers to money, etc). Better to tag it, and let the user decide.

      I was wondering this myself when I set up spamassassin on our mail server here (We're a web hosting provider using Communigate Pro). It filters mail for everyone by just changing the subject line to let them know that it's been marked as spam. From

  • Spam Assassin and/or Popfile (Score:5, Informative)

    by Universal Nerd ( 579391 ) on Thursday May 08, 2003 @04:29PM (#5913317)
    Ok, I'm not the email admin here at work, I avoid the whole mail subsystem 'cause I already have enough to do elsewhere.

    Anyway, for our 1500 users we use SpamAssassin with RBL and blacklists and our meager server (PIII 1.26GHz with 512Mb RAM) doesn't even reach 0.20, the heuristics is turned down due to the processor usage but it filters about 90% of the spam with very little load.

    I, personally, use Popfile (search Sourceforge) as my personal filter - with it's database right now, not that big, just some 8Mb with over 200,000 emails since training (from my huge spam database) and normal usage over the past year for me and a dozen other users. Very easy to set up and use, you just need to train it with a good database. It's stats state that it has a 99.85% correctness rate. The machine has reached .20 running Popfile during a monday morning after a long weekend but the machine is half the other one, a PIII 667Mhz with 256Mb RAM.
    • I'm also going to recommend POPfile, especially with the SMTP proxy coming in 0.19.

      I set up POPFile my last week at my old job. 20 users, 1000 emails a day, 80%+ of which was spam.

      PIII 1GHz, 512M of RAM running Win2K server. Load on that box despite running Exchange to boot was maybe 5-6% CPU when idling, and their accuracy only 2 months is 99.98% and we've never even reset the statistics!
      -----
    • I've been using Popfile too. It doesn't seem to slow things down any, downloading 3-400 messages a day over a cable modem (most of which is spam, and is successfully marked as such). FWIW I have a 700 Mhz machine, and use Win2k Pro w/ Outlook Express.

      Now that I'm looking to deal with all my mail from a server, I'm trying to find a way to use the Popfile filters I've so carefully trained over the last few months!

      BTW, my Popfile's accuracy is also just under 99%.
  • What you may need is loadbalancing and multiple servers. Granted, it's a function of how much mail on the whole you have to filter, some form of loadbalancing will be needed.

    Round robin dns'ing, a load balancing machine, a firewall that can do the likes (bigIP, yuck, i hate them).

    Your question is geared towards SMTP, but it's generally a network service question and how to handle X amount of traffic with Y resources.

  • Cloudmark Authority - a consensus based blocker (Score:3, Informative)

    by Cuchullain ( 25146 ) on Thursday May 08, 2003 @04:29PM (#5913322) Homepage
    I have had pretty decent results with the spamnet client on our mail clients. They are now using the data that the free clients generate to block at the server. The product is called Authority, or something like that.

    I receive about 50-70 spam mails per day, and the client has been blocking 98 percent of them every day. I have been very impressed by it.

    See if their server product is appropriate for you. It simply uses a consensus derived list from client users to block messages at the server. Kind of a blacklist thing.

    Cuchullain

  • The problem is hard (Score:5, Informative)

    by PD ( 9577 ) * <slashdotlinux@pdrap.org> on Thursday May 08, 2003 @04:36PM (#5913408) Homepage Journal
    Classifying spam is essentially the same problem as classifying programs into those that terminate that those that don't (the halting problem). This leads us to the following conclusions:

    1) Filtering spam is not trivial. A program that filters spam X% better than another program will be X^2% more complicated or worse.

    2) You can't write a program that will filter perfectly. At best, all you can do is develop a set of heuristics that you hope aren't too complicated. The less complicated the heuristic, the fewer resources it will require.

    3) There's a limit to how simple your heuristics can be.

    4) The system of spam is not just the message: it's the spammer, plus the message, plus the recipient. This is because a certain message considered spam by some will not be considered spam by others. That means that the heuristics that account for the person reading the spam will be better than those that don't. The source of a spam is also important: a message consisting of a spam report to a spam newsgroup is not a spam, though it may contain a complete spam message.

    5) The best spam filters will eventually be AI's that understand human language. That means that the ultime spam filter will require enough processing power to model human cognitive abilities. In short, you're going to see an endless increase in the number of processor cycles consumed by spam filters, asymptotically approaching the requirements of a full-up human brain simulation.

    On the other hand, this will sell a hell of a lot of computers.

    • Classifying spam is essentially the same problem as classifying programs into those that terminate that those that don't (the halting problem).

      What an amazing claim. Could you elaborate on this? For example, how do you apply Cantor's diagonalization argument to email?
      • Simple. Spams or programs can be represented as sequences of numbers, making them indistinguishable at the machine representation level. Take all of those bits of those numbers in a row and read them as a single number that represents that spam.

        At this point, Cantor's diagonalization is trivial.
        • At this point, Cantor's diagonalization is trivial.

          You have an interesting definition of "trivial".

          Ok, suppose I have a function which I claim distinguishes with perfect accuracy between spam and non-spam. How do you propose to construct a message which it mis-identifies?
          • Let's get real theoretical now: :-)

            Suppose you constructed a message that said in part: 'You have an interesting definition of "trivial".' You ran this through your perfect classification function and it said "NOT SPAM".

            I received the message and said to myself "well, look at that spam." Clearly the spam classifier mis-identified the spam as being ham, because I would consider it spam.

            I'm not trying to duck your question. OK, well maybe I am. I spent a bit of time thinking about it, and it seems that dia
            • That's my point. The problems with building a spam-classifier are practical, not theoretical. Until we agree upon exactly how we define "spam" we can't build a perfect classifier; but there is no analogue to the halting problem (Turing), the incompleteness-inconsistency theorem (Godel), or the diagonal argument (Cantor).
    • The best spam filters will eventually be AI's that understand human language. That means that the ultime spam filter will require enough processing power to model human cognitive abilities.

      That may not be true, and it goes to the same issue of definition of spam that makes this not an example of the halting problem. Since the final definition of spam is in the eye of the beholder, only a perfect model of a particular person's cognitive processes will be able to definitively distinguish spam from non-spam

  • I've been using these filters for quite some
    time, and they catch about 95 percent of it.

    Enjoy:

    If From/Sender or Subject contain:

    post-line, yahoo, mail, (your account name),
    postforme, photos, hot, degree, earthlink, aol,
    opt, cum, young, hollywood, notme, naked, penis,
    bigger, usa, model, women, girl, slut, prize, won
    msn, horny, dirty, gang, where, winner, price,
    teen, printer

    Move to folder spam. :)

    You'll get some false positives once in a great
    while, but it's nice to have all your spam in
    one folder and we
  • Have someone else do it for you. There are companies around that do mail filtering and so forth; they use Postini here. Costs a couple bucks per month per mailbox, but they also hold onto mail if the server here becomes unavailable, and it's never your headache to keep up with.

  • I've got a better solution (Score:3, Informative)

    by .@. ( 21735 ) on Thursday May 08, 2003 @04:47PM (#5913558) Homepage
    I've been thinking about this problem, and its various dead-end solutions (micropayments, rewriting SMTP, strong client/server auth, third-party circles of trust), and have come to the conclusion that none are necessary, or particularly desireable.

    I've put together the beginnings of an alternate proposal [bitshift.org], which draws on some of the good aspects of the above approaches, without the need to rewrite SMTP. It's a community-based, peer-based approach that leaves the power in the hands of the operator. Plus, there's no profit motive (except that it's in an operator's best interest, and thus the corporate owner's best interest, to maintain his/her server's level of trust).
  • mail.yahoo.com. :)
    • This can seemingly be improved upon. For reasons that are no longer clear to me, I have a yahoo mail account that does not match the username of the yahoo account. I get about 1 spam per month to this address; Of those, I think some of them are coming to a different address that I have forwarded to the yahoo account. Because of this, and use of another yahoo account as a spam trap, I have yet to experience the hell that many people seem to be going thru with spam. Add in something like YahooPops(check sf.ne

  • One very fast check is extremely effective: (Score:3, Insightful)

    by -dsr- ( 6188 ) on Thursday May 08, 2003 @04:59PM (#5913667) Homepage Journal
    One very fast check is extremely effective: look at the first line of each MIME attachment to see if it's a Microsoft executable file. If it is, quarantine it.

    (I wish I had thought of this, but Russell Nelson did.)

  • Bogofilter rocks! (Score:3, Informative)

    by bobv-pillars-net ( 97943 ) <bobvin@pillars.net> on Thursday May 08, 2003 @05:06PM (#5913735) Homepage Journal
    I had the same problem.

    Was pretty happy with spamassassin, but our mailserver was crumbling under the load.

    Switched to bogofilter [sourceforge.net] and, after a training period, we're now getting better accuracy (97.6%) with spam recognition than we did with SpamAssassin, with MUCH reduced server load.

  • I've used (Score:2, Informative)

    by motha_chucker ( 592192 )
    Xwall [dataenter.co.at] because we are running Exchange. It supports Bayesian filtering as well as MAPS/RBL rejection and virus scanning. Currently I am running only MAPS/RBL and have found it to be very effective with very few false positives. To answer your question regarding effectiveness of quick checks, I would have to say in my experience that they are effective. I have not stopped 100% of incoming spam but I would say around 98% and feel that is acceptable. Xwall is also cheap, $300.00 USD. Unfortunatley it will only
    • I bet you could support a couple of small central African villages for $300 a year...

      "When I was a boy, you could get a Baby Ruth bar for a nickel, and it was as big around as your leg."
      • Yes, for something that can support anything close to 50K users, $300 is way cheap. The OP said that he's using Exchange, if you're laying out for Exchange licenses, then again, $300 is way cheap.

        The "real world" sometimes requires you to actually spend money. Sometimes paying for things is cheaper than trying to piece together a bunch of other non-releated packages.

  • Multi level approach (Score:4, Insightful)

    by linuxwrangler ( 582055 ) on Thursday May 08, 2003 @05:18PM (#5913835)
    The more "low hanging fruit" you pick off the less your computationally expensive filters have to do. For example, if the other system greets you with:
    EHLO your.machine.ip.address
    or
    EHLO your.machine.name then it IS a spammer. Reject now. There are some patches and configurations for Postfix so you can declare that RCPT from certain domains like yahoo and hotmail be verified to have a hotmail EHLO that properly resolves. This is more expensive as a dns lookup is required but this will probably be cached locally pretty quickly.

    You can also unceremoniously drop any connection that starts pipelining before you say it is OK to pipeline and any EHLO that has an illegal hostname.

    This, at least, reduces the work your scanning engines will have to do. Still, even if you catch nearly all the spam with the easy checks you will only reduce your mail volume by ~40% (current estimated overall spam volume) so that leaves you with 60% to scan.

    I suppose your main MX could do the easy checks then send the remainder off to as many round-robin scanners as necessary which in turn could pass the mail on for delivery.

    One starts to realize why some places just roll over and pay tens of thousands of dollars to someone else to do it for them.
    • Do you know of any sites that explain how to do some of these for sendmail, either via patches or (preferable) using some config changes? Thanks.

    • You can also unceremoniously drop any connection that starts pipelining before you say it is OK to pipeline and any EHLO that has an illegal hostname.

      Dropping connections like this is not a good thing since the other party (ofc. depending on the implementation) will assume that due to network problems the connection failed; resulting in a re-connect after some time-out.
      This may effectively drain more resources than you were trying to save. Always send a 5xx return code (permanent error) to the server,

    • That's what I was going to suggest.

      I would start with a static domain-based blocking scheme. It requires a bit of maintenance (I need to add 10 or so domains/week), but I reject a LOT of mail with no false positives.

      Then use a more computationally intensive filter to catch what gets past the domain-based blocker. Potentially tie them together. (Have the computationally intensive checker make a list of domains. Then you can checkmark ones you want to block. I get legit mail from Yahoo users, so I can'

  • Run spamd/spamc version of SpamAssassin (Score:5, Interesting)

    by bill_mcgonigle ( 4333 ) on Thursday May 08, 2003 @05:20PM (#5913855) Homepage Journal
    SpamAssassin can run as a daemon (see here [spamassassin.org]) so it doesn't have to start up the perl interpreter for each message. This is the preferred mode for large installations.

    People report processing times in the range of 0.2 to 0.5 seconds per message with basic tests (no pyzor 2). Get a fast machine with dual processors, plenty of RAM, a caching DNS server, set spamd/spamc to have an appropriate number of child processes, and you should be good to go.

    It's certainly going to be cheaper than the sexual harassment lawsuit that one of those 50,000 users is going to file for being forced to look at pornographic material (we require employees to read their e-mail, don't you?).
  • Instead of having a centralized spam filter, why not have a spam filter on each users's individual machine? Sure spam will get through, but each user will take care of their own spam problem. This will also make it the users fault if something gets blocked. I mean it doesn't even have to be the best filtering in the world. Just use Mozilla Mail and the spam filter is has built in.

  • Tarproxy (Score:2, Informative)

    by blacksqr ( 187231 )
    Tarproxy (http://www.martiansoftware.com/tarproxy) seems like such an excellent solution, I don't know why it's not more visible. I would think that if just a few large mail servers started using it, spam might virtually stop overnight; thus rendering discussions of efficiency of filters moot.

    I can only think that commercial spam filtering companies are terrified of it, thus are somehow keeping it out of the public eye.

  • Answers (Score:1, Interesting)

    by Anonymous Coward
    You said you'd like to actually reject some mail. For this to work it has to be done during the SMTP transaction. You can't wait until the LDA gets its hands on the message. You have to do it at the MTA level. SpamAssassin can still do this. However now you need to glue it to Sendmail via a Milter. I highly recommend MIMEDefang [roaringpenguin.com]for your milter. Actually if you're rolling it out for 50,000 users then I recommend you purchase the commerical version called CanIt [canit.ca]. That way you get support and features t
  • I know it's an uneducated question, but would it be possible to put two (or more) spam filters/programs back to back? You could find some cheap and dirty filters, lower their spam threshold(s) to reduce/eliminate false positives and hopefully still stop a majority of the spam.

    is this too complicated to implement?

  • Our mail server is for a hosting company, it currently does about 10000 emails a day. The server itsekf is a quad PPro 200 with 512 megs of ram. Not only is the load below 1% all the time, our setup is more cpu intensive. Spamassassin is run out of procmail so it can be turned off for people who don't want it.
  • We're currently handling mail for 4k+ accounts using 2 frontend servers running postfix that do all of the filtering and then pass the messages back to our backend mail server.

    The frontend mail servers are running amavisd-new which is configured to use spamassassin and clamav. You can use DNS RR or just have multiple MX recs to load balance as many of these filtering servers as you need. Our filtering servers are cheap XP2100+s (w/1GB of ram) in a rack mount case that cost us ~$650 each. Amavis is just
    • I second this proposal. The amavisd-new+spamassassin combination is a highly efficient way to eliminate spam (rejecting the message if it's spam, as you requested) with near 100% accuracy.
    • I've myself been looking at antivirus solutions for our dormatory 400-500 userbase. We found out that it would be quite expensive (Sophos, HBedv and others), until we saw RAVantivirus. 300$ a year first year and then 80% off the next years to come (60$ a year). This is for a single mailserver with up to 5000 mailboxes, and it works perfectly, installs in 10 minuttes and has easy configuration. Look at www.ravantivirus.com

  • Filter via proxy, not LDA (Score:4, Insightful)

    by runswithd6s ( 65165 ) on Thursday May 08, 2003 @06:52PM (#5914668) Homepage
    Spam and virus filtering in an efficient manner for anyone is a major issue, and it has already been mentioned that there are multiple ways of accomplishing this. In designing your process, think in terms of dropping or rejecting email from the process loop as soon as possible.

    At the SMTP server

    • Drop email from known blacklisted servers via your email server access file
    • Allow email from known whitelisted servers or addresses
    • Use RBL lists
    • Filter out mis-behaving SMTP servers, ones that don't follow standard protocols
    • Disable ESMTP commands that give the spammer access to your local users lists (VRFY, etc..)
    • Only relay email from authenticated servers and users
    • Impose a size limit to messages (50k) if possible.

    At the SMTP Filter Proxy Server or LDA

    • Allow emails from recipient-based whitelists
    • Drop emails from recipient-based blacklists
    • Process Tagged messages (from TMDA)
    • Run your faster classification programs: clamav (for viruses), bogofilter (bayesian)
    • Run your slower classification programs: procmail, spamassassin

    Just remember to shortcut the process along the way. If email can be dropped or tagged for any reason, do so immediately and quit processing it.

    • 50K per message, are you INSANE??? I hate it when servers limit me to 2 or 5 MB let alone something so insane. Let me guess, you never send anything with attachments, get real.
      • "50K per message, are you INSANE??? I hate it when servers limit me to 2 or 5 MB let alone something so insane. Let me guess, you never send anything with attachments, get real."

        Why are you passing by value instead of passing by reference?

        That is to say, why are you sending an attachement instead of a URL or some other pointer to the file?

        Chewie did say "... if possible". That hardly sounds insane to me.

  • user education (Score:4, Interesting)

    by Parsec ( 1702 ) on Thursday May 08, 2003 @07:02PM (#5914748) Homepage Journal

    You could take some steps on the user education side of things. Before being given an account, they should learn a few things about how to keep their address safe, like:

    • spamgourmet.com and other disposable email address providers.
    • The ethics of buying from spammers (some people really don't know!) Make the counterpoint that it's perfectly acceptable to buy from sponsors of lists that they want to be subscribed to, to help support the list.
    • Always watch for checkboxes with tricky text used to gain permission when submitting their email somewhere.
    • When to click that unsubscribe link (which spam may be legitimate).
    • Offer to teach the finer points of tracking down and reporting spam. Report not just the sending IP, but also advertised web site, using the various web whois interfaces.
    • Point users to legislative possibilities that they may wish to contact their governmental representative to support.

    Also, if you're working for an organization which may want to expose user addresses to the internet via a web site, you may want to work with the web master and legal to create a click-through agreement that would stop spam harvesting robots while only requiring a couple extra clicks for the legitimate public. Or work with the web master to create a standard human-only readable way to post email addresses, e.g. "email lauren at our domain of example.com".

    You may wish to register an additional domain or two to provide disposable email address services to your users.

    Consider a piece of software that blocks IPs attempting to brute-force email addresses. Some filter monitoring the logs for excessive bounces from an IP and passing it to the firewall would work. I don't know of any examples of this software, but if you're doing a large email service you may get these kinds of attacks.

  • Does anyone have tips for sendmail configuration for some of these, eg disabling ESMTP user listing, blocking irregularly configured incoming SMTP servers, etc?
    • extract from a sendmail.mc :-)

      INPUT_MAIL_FILTER(`spamassassin', `S=local:/var/run/spamassassin.sock, F=, T=C:15m;S:4m;R:4m;E:10m')

      INPUT_MAIL_FILTER(`mimedefang', `S=unix:/var/spool/MIMEDefang/mimedefang.sock, F=T, T=S:60s;R:60s;E:5m')

      define(`confPRIVACY_FLAGS', `authwarnings,novrfy,noexpn,restrictqrun')dnl

  • i don't know too much about the real world of spam checking but it seems from the spam that i do revive much of it is redundant per domain. would it reduce the computation if you checked for redundant messages (mass spam messages) before you fed the stream of messages into the spam filter? --adam

  • SMTP rejecting of spam considered harmful (Score:3, Interesting)

    by Mozai ( 3547 ) on Friday May 09, 2003 @08:53AM (#5917803) Homepage
    Scanning messages for spam and rejecting at the SMTP level is a very bad idea. I'm the sysadmin for a company where about 25% of our email message traffic is spam. However, we also have a hard-working sales department who actually need commercial and sales messages. If a message from a client is marked as 'spam' because they're negotiating a sales deal, the sales staff still need to see this message. If a client's counter-offer is rejected at the mail server with a "you sent us SPAM" message, you can kiss that potential income goodbye.

    False positives can be more harmful than messages getting through the spam filter.
  • Ideally, I'd prefer something that does reject the message if it's spam (SMTP result code 550 or something like that), unlike current Spamassassin or spamprobe setups that accept the message and only later decide whether it's spam.

    If it's not directly aimed at you, you DO NOT delete or reject it. PERIOD. You tag it. Maybe you even quarentine it. But you DO NOT reject it out of hand.

  • Most of the "Bayesian" spam filtering is naive Bayesian, which is really just an important-sounding excuse for using a computationally expensive and simplistic classification method.

    If you want computationally efficient methods for detecting spam, look into decision trees (search on Google for decision trees and spam filtering). If you set them up properly, they result in a sequence of simple tests like "Is this addressed to me?", "Does the subject line contain the word 'penis' or 'breast'?", etc. Like

Slashdot Top Deals

On the eighth day, God created FORTRAN.

Close