What Ways Can Sites Handle Spambot Attacks?

Amazing Quantum Man asks: "I'm a member of a site devoted to nitpicking TV shows and movies. It has always had an open posting policy — no registration required, and you could use any name you wanted. This policy was instituted way back in 1998, and led to some quite fun, freewheeling threads on various boards. Recently, we have come under spambot attack, with spambots posting links to gambling and porn sites on every single discussion board on the site. The admins have been trying to block IPs, but it's useless against a botnet. As a defense, it looks like the site is going to require registration, and disable anonymous posting. Many regulars, while they understand the need, are concerned that the freewheeling character of the site will be lost. Let me continue by saying that I'm not a site admin, merely a member there. Also, if it helps, the site in question is running Discus. Has anyone here been in a similar situation? How did you handle it, and what did it do to the 'culture' of your site?"

Nothing is perfect

• Some sites use CAPTCHA... but I don't like it. I'll bet you I make a mistake in the CAPTCHA at least 30% of the time, which is just frustrating.
• Verified email accounts - this is what I tend to use. User signs up, email with password gets sent. Some people don't like giving out their email for fear of SPAM and such.
• Heavy user moderation - seems to work overall, look at /.

Re:Nothing is perfect

I have about 100 sites (really) and I've evolved with different methods. This is what worked with me.

First, when I identified what the spambots read, then I figured out how to fool them.
They read the form data; what the form posts to and what the form names are.
They populate the form names and posts to the action.

I removed all javascript validation. It's useless. Do 100% server side validation, verify email address are valid, links are valid, dates are valid, word count for submission, check for duplicate data for multiple form elements, etc...

I added session ID checks and this cut down on 75% of spamming where the sessionID is in a hidden field and if the request doesn't match the sessionID, it doesn't post.

I then separated the form from the page by using iframes.
On the initial load of the form, the proper HTTP REFERER is comitted in a session. If the form doesn't have the allowed referer, the form doesn't load and that form is blocked for the session with the IP address noted.
99% of the IP addresses are from China, Latin America, Russia, The Netherlands, and Africa.

Of the 25% of spam still coming through, I had to figure out the next step to stop it without compromising user functionaliy as in CAPTCHAS. There is no way I was going to use those nor use a `click the kitten` method either.

I rewrote the form code to change the form elements names for every load.
It was pretty much a hack but it worked.
I had a random 6 character word generated every load.
I dismantled that word every 2 characters and put 2 characters in every other character for the form element names that had been base64 encoded.
I had an empty hidden element that had to remain empty as well.
Bots tend to take every element and give it a value.

That seemed to get rid of the other 20%. After a while, the spam would continue at nowhere near the level it once was but we noticed that the timing was 5 minutes between replies instead of seconds meaning that the elements had to be filled out semi mechanically instead of automatically.

After copying that format for a number of forms, the spams that were coming through were from the same pool of networks.

After data crunching and some time, I realized that the obfuscating of element names really didn't deter much as much as sessionID and allowed refering pages did.

I started to actually have a single form for all like forms and use that one form for multiple sites so that updates can happen across all sites at the same time instead of updating 80 or so forms across sites.
I also am in the practice of banning IP address blocks for form access. If they really have something to say to us, they can contact us via email.

Email you say is probably the bane of existence for us that receive spam.
There are tons of javascript mail obfuscators and as long as you have a single email for mail contact, obfuscate it and only use that for mailto links.

I can seriouly attest that for the past 13 months, I've never received a penis enlargement mail at that address or any other stock tip.

My forms are hosted at a single location and have strict referer checking. Any attempt to `figure it out` by looking at the iframe source is banned.

If I get a form with non-relevant data, that IP is banned and all my sites and forms benefit.

I've gone from 300-400 form requests a day to the legitimate 10 valid responses a day

Idea

What do you think about need for registration and still keeping old open way of posting?
Just log in and later post with whatever nick you want. Just don't trace it or anything. You can even prepare some kind of statistics for users (how many post they posted). And of course implement some captcha.

No Registration Required

See if you can set up a CAPTCHA that must be completed before the post can be put up. Multiple missed attempts could even ban an IP. Just be sure you have some alternate means for people that have issues with their vision.

Why lock down membership?

It has always had an open posting policy -- no registration required, and you could use any name you wanted.

There's no reason why that should change. Just add CAPTCHAs of some sort or another to the posting system. No more bots posting crap (although the CAPTCHA system might need to be changed every now and then depending on the strength of those chosen).

Detect the incoming OS type

Amavisd-new has had p0f support for detecting the OS of the sending mail server for quite some time. It detects the OS type of the incoming mail connection and adds a header indicating the results. You can then use SpamAssassin to detect the OS an add an appropriate point total. Since few "real" organizations use desktop OSs for mail relays, you can usually assume a high probability of spamminess from such.

Akismet

Akismet [miphp.net] is a very good antispam. It blocks 99% of spam on my forum.

CAPTCHA doesn't work, many spambots can solve CAPTCHAs.

Captchas

I would suggest maybe putting in Captchas [wikipedia.org] for every spot you might submit a post, etc. This way, bots cannot or have more difficulty making posts. Here are more links I had on these, but I haven't looked at them in a while...

• www.kittenauth.com [kittenauth.com]
• www.hotscripts.com : captchas generator [hotscripts.com]
• Slashdot [slashdot.org]

links

Don't let anonymous users post links to other websites.

Simpler than Captchas

On my guestbook, I just say "posts must begin with the word Banana, which will be automatically stripped." It works. Some spambots are actually human so it doesn't stop them, but it's super-simple.

Lame ideas from a tiny site

I used to get a ton of spam on my guestbook. I tried doing lots of little things in the code and it turns out the spam was being submitted without them filling out the HTML form. To force this to happen, I found a neat idea on some German website (it's down, so I mirrored it [princeton.edu]). The code will not accept a post if there is no number/checksum pair.
That cut out a lot of the spam. The rest has been gone since I added another, required field "What is my first name?" It is like a captcha but much easier. No one will complain that they get it wrong. For your site, maybe something like "Finish the name of this show 'I Love ...' "

Use mod-security and Bad-Behaviour

Use Bad-Behaviour [homelandstupidity.us] and mod-security [modsecurity.org].

These two work perfect for me.

The never-ending battle

I've been battling this for years now. Ironically, the best way to stop spambot attacks is to homebrew your own CGI stuff. If you can't do that, rename all the scripts to non-standard names so that the common URLs are not found.

I've been using keyword blacklists. They have proven to be very effective. If you don't allow people to input names of common drugs or strings like ".php?" or ".asp?" you can knock out a lot of the affiliate/redirect spam.

The biggest problems have been with the popular messageboard apps. We've simply stopped putting up messageboards, or set them to require registration and manual approval to post. It's really disgusting how if you leave a forum unlocked, it'll take about a week before it's full of ads for online drugs and sex sites.

CAPTCHA

I've had to deal with spam attacks on both my personal site and a forum I use. In both cases, we tried to ban IP addresses, then tried invisible methods of stopping spam (eg hidden required fields populated by javascript), and nothing worked.
In the end in both cases, we've just had to use a CAPTCHA system. Spammers tend to use multiple IP addresses (and I do mean in the hundreds, a lot of them proxies or botnet-controlled boxes) so banning simply doesn't work.
I've tried doing things like only requiring a CAPTCHA if the comment includes "http" or similar techniques. It doesn't work, I've had spam that simply consists of "Hi, great site" posted 30 times.
I don't know why, but spammers don't seem to care whether their spam even has the potential to turn into revenue for them or not..
CAPTCHA is the only viable method, IMHO.
For those worried about accessibility; offer a non-CAPTCHA'ed form and manually review it; most users will be able to post perfectly well and for the few that can't enter the CAPTCHA, they can still post to the site, but with a delay as you check it for spam.

Approve once, post always

On Wordpress [wordpress.org] you have the option of requiring moderation only the first time an individual posts. Once you have approved one post by them they no longer are moderated.

Sure it still involves trolling though moderated spam to find the genuine posts, but if you don't have massive traffic it works fine.

Need an HTTP greylist similar to OpenBSD's spamd

After seeing this presentation on OpenBSD's spamd [ualberta.ca], which profiles and greylists SMTP connections coming from botnets, I'm convinced of the need for HTTP POST greylisting.

Point is twofold: slow the bots down (or stop the dumb ones altogether) and block obvious botnets completely.

SMTP has the handy retry message. For HTTP, we would need to store the original POST request, and return a response with a 10-20 second meta-refresh to a confirmation url. Anonymous posters won't mind the wait, and the time window gives us time to watch for additional POSTs from the same ip, and blacklist them outright if they match a spammy profile.

how about-

registration required to post a URL or email address?

This is an easy one...

Check your users against DNSBLs [wikipedia.org]. Originally intended to block out malicious mailservers via their IP addresses, they are applicable on webservers as well. Via sorbs [sorbs.net] you can check for open HTTP and SOCKS proxies (interresting for you), open SMTP servers (not very interresting for you), webservers with unpached vulnerabilities, hijacked IP netblocks and malicious (in bed with spammers) network service providers. Other lists include the here recently mentioned Spamhaus [spamhaus.org.uk] list, and various DULs (dial up user lists). See the Wikipedia article for some of them.

I used DNSBLs at my former employer to block users coming through open proxys from registering domains. We saw that every phisher who bought a domain name came through an open HTTP proxy and used a stolen credit card. So using DNSBLs was the only viable option then.

One way I block spam...

I have successfully blocked comment spam by rejecting messages with http:/// [http] in them. Most of the spam contains links, so this can be extremely effective. Maybe on the site in question, reject anonymous posts that have http links in them, and if you have a site you need to post, you have to get an account.

Sean

Spambots

I've had good success with grep(1), using a file filled with various words culled from spam.
I also recycle known spam through the search software, so it automagically updates itself. Seems to work well, and the
best part is that as your anti-spam technology improves, the people behind the spam robots tend to give up on your site.

Spammers are dirty creatures

I dealt with this same issue on a message board. For years it did not require registration to post and with a small cadre of level-headed moderators we had a lot of fun. It was good for everybody, from regulars to one time guests who just wanted to ask a question.

Then, about two years ago (I think), the message board spammers began to get exponentially worse. Poker spammers were most of it, but I also saw a number of porn site spammers and some guerilla marketing campaigns that were awful. The evening that the one "documentary" on M. Night Shyamalan played on SciFi a huge number of posts and threads from "people who watched the film and wanted to talk about it" appeared. Obviously a bot network, because there were easily a hundred posts and the IP addresses were checking out as valid.

I tried everything to avoid registration. Banning IPs was useless, because they were bot networks. I made rules to discard posts that matched known spams - new, different ones came in. I discarded multiple posts or duplicate posts - the bots made posts that were different. I made rules to discard posts with certain URLs - no good, way too many URLs were rolling in. I changed the name of post function files in the Phorum message board - the bots adapted or were adapted. I made rules to prevent multiple posts within a certain period by the same host - the bots slowed down their posting. They posted with http code, they posted with bbs code, they posted plain text. In the end, after about two months of too much effort, I enforced registration. The problem has been solved ever since.

As a result of the registration I am certain my message board is not as robust as it once was. The simple fact is that registration drives away people who could become good members of the community. Another simple fact is that I have seen a number of boards turned into useless crap by spambots.

I dislike CAPTCHA, so registration was the lesser of two evils. However, if there is a mod so that Phorum can enforce CAPTCHA for guests, thus allowing them to post without registration, maybe I should check it out.

Ours works just fine.

I've been active for quite some time on a site dedicated to DIY tube guitar amps (ax84.com). We have a lounge area where anything goes, but the posting policy is quite loose, with all sorts of fun stuff occuring within [otherwise] on-topic threads as well.

After getting hit with several posts by auto-spammers, the maintainer instituted new rules.

You can register, which requires nothing more than a valid email address, handle and password (AFAIK, I registered when he was first testing logins). But we also have people who don't want to register for a variety of reasons-- from wanting to stay off the grid to just not caring. These people get a temporary login if they answer a question that is easy for humans, less easy for a bot. It could just as easily be a "pick the number from the image" thing or whatever.

At any rate this has been in place for a month or so, and I don't see any difference at all in the community. It's still a free-wheeling, fun place, but no spam. A win-win from where I sit. It's possible the non-registerers are unhappy, but since Chris included them in the discussions of how to handle things, and they are still there, I have to assume they're "happy enough". I am.

Re:Use an email verification

Won't your database get full of posts that were never verified? I suppose you could auto-expire them.