Slashdot Log In
Smart Spam Filtering For Forums and Blogs?
Posted by
timothy
on Sun Dec 28, 2008 04:48 PM
from the world-will-beat-a-path-to-your-door dept.
from the world-will-beat-a-path-to-your-door dept.
phorm writes "While filtering for spam on email and other related mediums seems to be fairly productive, there is a growing issue with spam on forums, message-boards, blogs, and other such sites. In many cases, sites use prevention methods such as captchas or question-answer values to try and restrict input to human-only visitors. However, even with such safeguards — and especially with most forms of captcha being cracked fairly often these days — it seems that spammers are becoming an increasing nuisance in this regard. While searching for plugins or extensions to spamassassin etc I have had little luck finding anything not tied into the email framework. Google searches for PHP-based spam filtering tends to come up with mostly commercial and/or more email-related filters. Does anyone know of a good system for filtering spam in general messages? Preferably such a system would be FOSS, and something with a daemon component (accessible by port or socket) to offer quick response-times."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Akismet (Score:5, Informative)
Second that! (Score:5, Informative)
Akismet [akismet.com] is the best thing for blog spam prevention ever. I can't believe you've never stumbled across it before. It uses statistical analysis to identify spam, and the more people use it, the better it gets. If everyone used it, the blog spammers would just disappear because their attacks would be completely ineffective.
Parent
Re:Second that! (Score:5, Informative)
Parent
Better than Askimet? (Score:4, Interesting)
Arguably, it is Mollom. Especially if you are using Drupal.
Askimet is 'rotting on th evine' in many ways - including development updates. Mollom is a commercial web service, with a free version for non-profit and small volume sites/users.
The Drupal module is explained here:
http://drupal.org/project/mollom [drupal.org]
The Mollom site:
http://mollom.com/ [mollom.com]
Parent
Re:Better than Askimet? (Score:4, Informative)
read my sig
Parent
Re:Better than Askimet? (Score:5, Insightful)
read my sig
That'll work, right up until the spam bots are told to ignore spampoison.com, or the person who is running the spam bots decides to put spampoison.com into his hosts file and point it to 127.0.0.1.
Lame solution.
Parent
Re: (Score:3, Insightful)
The problem with that concept is spammers just have to "blacklist" spampoison.com. Or implement "spam filters" of their own to detect such site
What would really be ideal would be thousands of poison domains, with high variability so smart spammers can't easily protect themselves and sanitize their lists, when they figure out what's going on...
Re:Better than Askimet? (Score:5, Informative)
I seem to get Mollum captchas on every site that uses it. My IP, user agent, etc. are almost completely static. My comments are grammatically correct, never spammy, etc.
If their system hasn't identified me as safe by now, there's something wrong.
In contrast, to my knowledge Akismet has never flagged me. My comments go straight up on blogs using it. On my personal site, I've had maybe 10 false positives out of several thousand caught.
Mollom, IMO, has a long way to go.
Parent
Re:Second that! (Score:5, Interesting)
Parent
Re:Second that! (Score:5, Informative)
I've used it for a few years now. In that time, it has caught tens of thousands of spam comments. It has missed about ten spam comments (i.e. allowed them through). It has misidentified two legitimate comments as spam. Yes, I realise I'm keeping an eye on it, and someone who doesn't may not notice that it's causing problems for them. But the stats are pretty good in my case. I'm aware of the allegations of corruption and using it to gag people, but that hasn't affected me yet.
Parent
Re:Second that! (Score:5, Informative)
I just noticed a handy Akismet stats link in the latest version. I've been running Akismet since October 2006, in that time there have been 26,575 comments on my blog, of which 26,302 were spam(!). It missed 25 spam comments that had to be manually moderated and passed 273 legit comments. There have been no false positives. Personally, I think that's a pretty darn good record.
Parent
Re: (Score:3, Funny)
My question is: If I can't read them, am i a robot?
well, let's see. would you injure a human being or, through inaction, allow a human being to come to harm?
I always thought (Score:2, Informative)
Re-Captcha was fairly effective and easy to install and useful.
D.I.Y. (Score:2, Insightful)
Or am I misunderstanding what FOSS really is about?
Re:D.I.Y. (Score:4, Informative)
Parent
Re: (Score:2, Insightful)
Re: (Score:2)
Or am I misunderstanding what FOSS really is about?
If your first instinct is to build it yourself, then yes, you are kind of missing what FOSS is really about. To jointly improve shared solutions, you first have to find the solutions that are already out there.
Re: (Score:2)
For one large site I took an open source Bayesian filter and customized it. This site was large enough to get spam that's only posted there, so a DIY Bayesian filter worked extremely well. They have staff to remove spam and illegal content, so the filter simply aided the staff, who were able to train the filter very quickly.
However, this solution would be useless without enough content and without people properly training the filter. If you get generic spam in a common scenario than a more generic soluti
Service.. (Score:2)
Various ideas (Score:2)
There are a number of things you can try:
For a small site I helped set up, they went to complete SSL and client certificates, where users had to obtain a cert from Verisign or Comodo before they would get access. This stopped spam, and one can obtain a client cert for free or a low cost. However, this can't be done for most forums or blogs.
For larger sites, a lot have ended up moving to an approval type of system where a human approves the creation of the user, then a limit on how many posts a first time
mollom not so free (Score:3, Interesting)
mollom [mollom.com]
i discovered this one through drupal. I thought it was completely free but apparently for high traffic sites it isn't.
I think all your user generated content is sent to them and checked for spaminess against the other submissions they are receiving and they give you back a rating.
Re: (Score:2)
I use Mollom because of its excellent integration with Drupal. It's free for up to 100 legitimate posts per day, 30 euros/month after that.
It works very well for stopping spambots without annoying real humans (which a plain captcha will do).
Human spammers still slip through, but when you delete their work, it's fed back to the Mollom database, protecting you and others from repeats.
DIY or it will be broken (Score:5, Interesting)
Any method you use can be broken. Your only chance is to reduce the likelihood that your site is worth the effort.
Basically, if you use a common solution - no matter of FOSS or commercial - then there will be a thousand other sites that use it too. This attracts attackers because they know when they hack it once, they can re-use it.
However, if you handcode something, no matter how primitive, it likely lasts a lot longer because nobody bothers hacking into your site...
Of course that doesn't work if you have a large site like myspace - there, a single site is worth the effort by itself.
Anyway - then there are two things - a really fast moving animated gif and silly things where you ask people to identify items usually work.
I help out with a site that randomly takes five pictures of cats and dogs and it asks you to identify which of the images contains the highest number of kittens... We barely ever get spam through - and that with almost 20K attempted submissions by non-humans a day makes us pretty happy
Peter.
Re:DIY or it will be broken (Score:5, Interesting)
However, if you handcode something, no matter how primitive, it likely lasts a lot longer because nobody bothers hacking into your site...
Simply renaming the .php files worked 100% for me.
Parent
Re:DIY or it will be broken (Score:5, Informative)
I have a very simple, small site that I run that allows small comments. It was fine until the spam bots found it. Anyways, I just added a simple question about the background color of the site, which must be correct in order for the comment to be posted. I haven't had a single issue since (except for the occasional troll, but what can you do about that).
The nice thing about something like this, a handmade thing, is that the spammers won't bother 'breaking' it. As the parent mentions, the spammers are attacking the common solutions - so a little home grown bit will work wonders.
Parent
4 Tests Stopped 30,000 Comments For Me (Score:5, Interesting)
Test one is, does the last name = the first name. For some reason almost all spammers do this.
Second, do they use a keyword from a list of about 15 words.
Third, do they fill out a hidden inputbox? This is sort of the reverse captcha.
Finally do they use more than 4 "http" in a post. Almost all comment spam is an SEO effort to increase their pagerank.
Hidden Input Box (Score:5, Informative)
This is really a very good test. As others have mentioned in this thread, it's the sort of thing that spammers will circumvent if it becomes widespread, but for now it's great.
There's something else I've found to be really quite effective: deliberately misnaming my form fields. For instance, give the input field that's labelled "First Name" an input name of "phone number." Humans don't use input names to determine what text to enter, but spambots do. Then check that inputâ"if the first name field contains a phone number, you know you've got yourself spammer.
I've used solely the combination of these two things to run one of my websites for two years now, and I get a vanishingly small amount of spam.
Parent
Bad Idea (Score:5, Insightful)
As someone who once used text browsers, I can only advise everyone not to do this - it breaks accessibility at a fundamental level: I got banned from a forum once because they mislabeled fields.
What however, works really great for comment spam is a simple question like "What is the name of Barack Obama ?".
Parent
My 3 tests also work (Score:5, Interesting)
My rules are:
1) The text boxes for things like name and subject are actually called junk.
2) There are hidden textboxes called name and subject (1 hidden by javascript and one by CSS) that if they are populated the post is ignored.
3) A third hidden field is the result of a simple javascript math equation that is checked on the server side. If the value is wrong, the post is thrown out.
As others have said, if your site is small these types of things are good enough to prevent spam because the spammers won't bother to figure it out. These concepts would never work for any of the larger sites or 3rd party forum software.
Parent
Re:My 3 tests also work (Score:4, Informative)
Parent
Re:4 Tests Stopped 30,000 Comments For Me (Score:5, Interesting)
Background: One of my sites is a custom job which kills a spam comment every 3 seconds or so, and has done so consistently for the past four years.
OP's suggestions are very good, especially limiting the number of 'http's. We've given up on the keyword lists since they are costly to maintain and aren't as effective as some other methods.
Currently, the most effective kill rules for us are:
1) We write the client's IP address, the ID of the thing being commented on, and random stuff to a cookie from the legitimate page from which the client clicked the "post reply" link. If the IP address doesn't match, or if the ID missing, or if the parameter for the random junk aren't in the cookie, then fail. This rule traps non-browser scripts and limits spam throughput, but does not affect humans.
2) The client's IP address is a hidden form variable. If that IP address does not match the IP from which the POST originates, fail. This rule traps the browser-based scripts, and operators who proxy through botnets for testing.
These two rules catch all but about two spam-like messages a month (spam operator not using proxies to test their scripts), and have mislabeled two legitimate messages (from a local ISP's poorly-configured proxy) in the last three years.
There are other things at play, such as salted hashes of the above, and some other heuristics on hidden and unused fields which sort and categorise the spam for our own research (including point of origin, topic, etc.). One finding is that IP/geographic blacklists are ineffective. I'll post new findings and methods in another two years.
I'm also evil in that the apparent failure modes are non-deterministic, and include such things as random HTTP response codes, random modes of connection failure, and spam messages that apparently go through, but are only visible for the IP that posted them, or for one minute after they are posted.
Your move, "RosarioRush".
Parent
Re: (Score:3, Interesting)
HTTPBL (Score:2, Interesting)
Project Honeypot's HTTPBL has been good to me:
See: www.projecthoneypot.org/httpbl.php
Fast way is to slow it down (Score:4, Interesting)
The fastest way is probably to just slow down user registration. Permit anonymous posting, but make it moderated/screened by default (ie. not visible to other users until the forum owner flags it as OK). When a user goes to register (so they can get their posts visible immediately), do not send them the confirmation e-mail immediately. Batch your confirmations up and send them out twice a day at odd times (ie. not midnight and noon, something like 3:47am and 3:47 pm) (you could do it 4 times a day, but not much faster than that since the idea's to introduce a delay in the registration process). Make sure to tell the user on the registration screen what sort of time-frame they can expect their confirmation to arrive in. Ordinary users who plan on using the forum long-term won't be inconvenienced much by this. Spammers... won't tolerate the delay, they want to get their message in fast and get out. With their automated scripts they might not even notice things are failing. Also, don't include a direct confirmation link in the e-mail. Include a URL to a form and make the user copy-and-paste the confirmation number from the e-mail. That'll be trivial for humans, but not easy for an automated script to handle without human assistance.
None of that will stop a determined spammer, but most of them are more interested in volume than anything else and they won't bother spending time/effort on just one forum when they could hit 10 others instead.
Re: (Score:3, Insightful)
Internet marketing brought us forum spam in the first place, I'm afraid.
For most forums it's not about getting the most users at any cost. It's about getting the most interested visitors without scaring an unacceptable fraction away, while at the same time keeping the number of spammers at a manageable level (which, given their proclivities, is pretty close to zero). And the simple fact is that, if it can be automated, spammers can and will automate it. And as long as it costs them little or no time or effo
YAWASP for wordpress (Score:3, Informative)
"I am a robot" field (Score:5, Informative)
It's got a field that says "I am a robot" checked off by default. A human should obviously see that and uncheck it. Those registrations that come in with it checked are blackholed. It's definitely cut down on the SPAM accounts since they enabled it.
Re:"I am a robot" field (Score:5, Funny)
Great idea, and it has the side-effect of keeping idiots out too :)
Parent
Re:"I am a robot" field (Score:5, Funny)
And the robots. Here I am, brain the size of a planet, and I keep getting banned from forums. *sigh*
Parent
If you're willing to pay... (Score:2)
Message board spam. (Score:5, Informative)
I had a similar problem in the comments area of my site. It was all fun and games, until one day I checked, and there were something like 1000 spams for every real message.
I wrote my own system to deal with it. It's not very hard, assuming you know how your site works (of course you do, right?)
I ended up making two blacklists. One was for words and phrases. The spammers tend to post (and repost, and repost) the same crap. My blacklist rules had some simple regular expressions that I could run queries with. Like, "%http://%spamsite%" and "%v%gra%". You get the idea. The second list was IP's that were known spammers.
At the time, I allowed both anonymous comments, and comments from logged in users. I eventually did away with the anonymous comments, as they were a headache. This was the best cure.
So, when my script ran (once a minute), if it matched a message, it would delete the message, and append the IP to the IP blacklist. If it was posted by a user account, the user account got suspended, so they could no longer log in, nor post.
After it's detection and cleanup run, it then ran back over the IP list, and pruned out every post by that IP. Sometimes they'll do practice runs saying silly things like "nice site". I thought they were real user complements at first, until I saw the same posting verbatim coming from the same IP to multiple news stories, and then that IP would start spamming later.
Some people will argue that the IP cleanup run was not nice, polite, or even fair. People use proxies. Sure, they do. We got a lot of abuse from anonymous proxies, and no real messages from them. The spammers didn't seem to like to use AOL.
When I implemented this, I posted a very brief description of what I was starting ("We're starting advanced anti-spam protection"), with an apology for real messages that were deleted. I never received one complaint about real comments disappearing.
How brutally you do it is really up to you. I built my method by manually doing it for a while, and then letting the script do it on it's own. Occasionally, I would have to go in and add new words and/or site names to the words blacklist.
I noticed the spammers hit more common software more often. It's worth it for them to make automated systems to abuse a piece of software that's deployed on tens of thousands of sites. When I rewrote my site from scratch, then abuses dropped down to 0 for a long time. Now, they manually submit "news" items which are just ads for their own sites. It appears to be manual, and since we won't run them as news stories (our editorial staff decides what does or doesn't show up as news, and if it needs to be edited first), they give up pretty quickly.
honeypot (Score:2)
I like honeypot links that blacklist anyone who clicks it. Seems to take out spam spiderbots effectively, until they learn how to avoid the honeypot links.
Alternative to Akismet (Score:2)
TypePad antispam is a great alternative to Akismet.
Pivot open source blogging (Score:3, Interesting)
The comment- and trackback-spam blocking techniques in Pivot blogging software are, from my limited personal experience, 100% effective. There's even an extension that uses the enormous Project Honeypot database (http:BL) to weed out IP addresses of identified harvesters and comment spammers. That's just for entertainment, though, since the basic techniques are completely effective.
Re:the solution is here .. (Score:4, Funny)
Yea, design an email system that is immune to spam and make the ISPs responsible for blocking spam, phishing and such attacks ..
So, I know people sometimes DRTFA. It happens. Life is busy. But, you know, it's always good to RTFS because it has fancy little tidbits of information such as:
While searching for plugins or extensions to spamassassin etc I have had little luck finding anything not tied into the email framework.
Parent
Re:the solution is here .. (Score:4, Insightful)
Parent
Re: (Score:3, Interesting)
That's a non-issue.
Want to block a ton of spam? Reject any inbound smtp connections that have no reverse DNS record, then use regular expressions on those that do to refuse connections from dynamic/home/dsl/dial_up/etc. (I tried to post the regexes, but slashdot whined about " Lameness filter encountered. Post aborted!")
Stop talking to dynamic IPs
Re: (Score:2)
Re:gmail (Score:5, Insightful)
"Do not allow registrations with gmail.com email addresses"
That is one of the most stupid things I heard this year.
Parent
Re:gmail (Score:4, Informative)
It's no less trivial than getting a Hotmail account, a Yahoo! account, or any of the many thousands of free webmail providers out there.
Even so, I suspect that the majority of casual Internet users today actually have that sort of email account, based on personal experience. If you start blocking them, you're blocking most legit users, too. Unless it's a technical forum - and even in this case it's silly to block GMail, as many techies use that.
Why? I for one don't have one - I use my GMail one everywhere - and I contribute to a lot of forums.
Translate, please. Explicitly email what where, and how is that going to help?
Parent