Coping With 1 Million SSH Authentication Failures? 497
An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"
sshblack (Score:2, Insightful)
sshblack works very well for me. It monitors failed ssh connections and inserts entries into iptables to block the IPs.
Exactly (Score:0, Insightful)
People are just paranoid. "How can I stop people from driving past my house and seeing if the lights are on?" You can't, cheaply.
An iptables recipie (Score:5, Insightful)
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j LOG --log-prefix "SSH_brute_force "
-A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j DROP
Stops 'em *somewhat* dead. If you want to whitelist hosts so they are not impacted by this rule, just add;
-A INPUT -s your.ip.address -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
Before the throttling ruleset.
OpenBSD PF (Score:4, Insightful)
you know what I am saying?
Re:whatcouldposiblygowrong (Score:2, Insightful)
If somebody tries to perform surgery without a proper medical education, do you offer pointers, or do you tell them to drop the knife and find a real surgeon?
Companies running public-facing servers without good technical expertise are a menace.
Re:Tar Pitting (Score:5, Insightful)
Let me see if I have this right. If iptables can keep count of incoming connections within the kernel and drop incoming connections as they happen without any intervention from a userspace program, that's _not_ automatic.
But running a gigantic shell script to scrape text messages out of your system logs and then call randomly chosen commands which may or may not have any effect on the observed problem some time after it occurs, then that's "doing that stuff automatically"?
Maybe those words don't mean what one of us thinks they do.
Re:Tar Pitting (Score:2, Insightful)
Re:Tar Pitting (Score:5, Insightful)
That, and create only a single user who can log in, that takes you to the real log in prompt. That way an attacker has to guess the one user+password, and have a legitimate userid+password to gain access.
It's not foolproof, but it foils the vast majority of script attackers.
And DISABLE ROOT LOGIN!
Re:whatcouldposiblygowrong (Score:5, Insightful)
That's a beautiful and witty analogy but to reverse it:
If he asked "How can I stop my house being burgled?" and you said, "Without a full alarm, CCTV and patrol system, don't even bother. Leave it to the professionals."
Some others might see some practical value in suggesting, "Maybe lock your door at night."
My opinion in this situation is first to take what advice you can and do something yourself as soon as possible (since any extra security is better than nothing). Next, as you suggested and when he has the resources to employ someone to do it properly, seek professional help. Simply doing nothing until he can do it properly sounds a bit dangerous.
Re:whatcouldposiblygowrong (Score:3, Insightful)
Since it seems to have escaped your attention, I'd like to point out the fact that unqualified surgeons can (will ?) kill people; unqualified admins won't.
Which seems to be a somewhat relevant matter.
Re:Ignore it? (Score:5, Insightful)
1 million per year over 50 sites == 20,000 per year per site, or 54 per day per site. Change the passwords every few months (and use different ones for each site).
Further - 1,200 different remote hosts means what, 17 attempts per remote host per site per year. Probably randomly p0wned PCs that hit addresses at random, make a few attempts using default or ocmmon passwords, then move on.
Re:Tar Pitting (Score:1, Insightful)
That's asking for a dos attack though, send thousands of connection attempts to port 22 and the server will end up holding them for eternity until port/connection exhaustion.
Re:whatcouldposiblygowrong (Score:3, Insightful)
People without perspective are a menace.
Re:fail2ban (Score:5, Insightful)
This is really cool until you find yourself trying to log in from the same access point where somebody with a virus was attached earlier in the day. Better to just use crypto (key-based authentication only) and rate-limiting.
Re:whatcouldposiblygowrong (Score:2, Insightful)
To whom, exactly, is it more important?
Society places economic value on everything, and the amount of damage a cracked server farm can potentially do far outstrips the economic value of a human life.
Re:whatcouldposiblygowrong (Score:3, Insightful)
Practically speaking, it really isn't different.
The surgeon who kills someone might, if the victim's family is very lucky, be assessed damages of a couple million. An idiot with a cracked server farm can easily cause that much damage on the net.
WTF? In one case, a web server has been vandalized. In the other, a person is dead.
Re:whatcouldposiblygowrong (Score:3, Insightful)
I'm sorry you're too psychologically immature to recognize economic reality, but that's really not my problem, and hardly makes me a psychopath.
It's not a sign of being "psychologically mature" to hold the notion that human life is a fungible commodity.
A poor person in the slums of Bangalore is economically worth more dead (for his organs) than alive. You state that society places an economic value on everything, including people. Yet for some reason, society has outlawed murder, even if the result is a net monetary profit for society.
Your complete failure to comprehend that a human life is worth more than its economic value to society is why I called you a psychopath.
Re:whatcouldposiblygowrong (Score:3, Insightful)
Re:hashlimit (Score:3, Insightful)
That's probably the most relevant thing I've seen. I'm amazed at how many places don't want or have people with a clue working on their equipment. TFA said they have *NO* sys/net admin, and the guy who attends to those task is a "n00b".
There's more than one way to address this (which plenty of us have mentioned), but the important part is to have someone on staff who actually knows what they're doing.
I had someone ask me the same thing a while back. It wasn't tallied over a year, but it was something like 10,000 connections from China to SSH on port 22 in an hour. They were all unsuccessful. I gave a few options. Move SSH. Bring up proper iptables rules (or ANY iptables rules). Put a firewall out front. Set rules in the router to prevent it. Their final decision was .... [drum roll] ... "well, as long as they were unsuccessful, it doesn't matter." That from a group who had a few machines compromised in the past for various reasons.
Sometimes it doesn't matter if you've given them the best, or even reasonable answers, they'll only hear what they want to.
Re:Move to a higher order port and use denyhosts (Score:3, Insightful)
You are assuming they are actually attacking your computer.
They are not. They're stupid bots. Poorly written stupid bots.
They do not do portscans, at least not of random ports. They do not attempt to find ssh elsewhere.
I swear, I keep having this idiotic discussion. Have it about spam too. 'But why would you block email servers that don't appear to understand the SMTP protocol? Spammers can just write better servers.'
We're not trying to block hypothetical attacks. We're not trying to stop hypothetical tools that go out and find ssh servers.
We're trying to stop the actual attacks that are actually happening from totally idiotic craptastic software criminals are currently using.
Forcing them, at the very least, to high competent amoral programmers, instead of the incompetent boobs they apparently hire currently.
Re:Move to a higher order port and use denyhosts (Score:3, Insightful)
I used to feel the same way. But really, moving the port lets you focus on the real threats.
Look at it this way: if you are being targeted with a sophisticated hack, any dangerous attempts to log in to port 22 will get lost in the massive amount of bot traffic. But if your SSH server is on port 12344, then the reports in your logs will be more meaningful. You'll know that someone was conducting real reconnaissance and you can start worrying. There's no reason that you can't change the port as well as requiring keys to log in.
Relying on a secret port is security through obscurity. Utilizing a non-standard port to filter out bot traffic in your logs logs so that you know when you're really being attacked is actually fairly smart. Sure, maybe one day the bots will start port-scanning first, but for now, it's sound.
Re:fail2ban (Score:3, Insightful)
Because stealing RSA/DSA SSH keys is so much more common than dictionary/brute attacks...
Lock it down. (Score:3, Insightful)
I can't believe the person who set the server up left this port open to the world. If you want the server secure then you will close all the ports that are not actually needed - in your case this likely means you keep 80 and 443 open.
For remote management make get a few IPs for the desktops that are meant to connect to it and make sure the ISP opens SSH port only to the IPs.
You can tunnel X over SSH and do file management, whatever you need so it should be enough.
Don't bother changing it to some random port, security through obscurity is total bullshit in this age of port scanners.
good against 1 attacking host -- not large botnets (Score:3, Insightful)
I am using fail2ban, but I do not think it's a particularly great tool when you see how many different IP addresses the attacks come from, so you're somewhere stuck in the middle trying to optimise how to make fail2ban more effective, while not being a problem for your own users:
- you configure blocking from 1 login attempt and a long block time cuts down on most of the outside attacks, but if some legitimate user mistypes his password, he'll be locked out for that same long block time...
- you configure several wrong password attempts and/or shorter blocking times - then the attacks do not get slowed quite as much, but it hinders your users less.
There are some things you might consider, though -
- if you can, and your users always access the system from their own computers/laptops, force the use of public-key authentication and disable use of password logins; then outside access attempts via password cannot succeed.
- if you can't do this, at least force root to use public key (and/or su/sudo from another login. (sshd_config either
'PermitRootLogin without-password' (to allow root access through authorized_keys, but not through password, or
'PermitRootLogin no' to block root access via ssh-login altogether.
- if there are only specific accounts you might want to allow access to, look into pam_require, then limit to those users in pam's ssh
config: 'account required pam_require.so @ssh-users myaccount' (allows user 'myaccount' and all users in group 'ssh-users'
to log in via ssh - everyone else will be kicked out, even with a valid password).
- keep an eye on the log to see whether it's actual users that get hit, or whether you see access attempts to non-existent users
(many of the attacks are basically dictionary based). If you find one of the valid accounts gets hit too much, consider renaming that
user account.
Re:hashlimit (Score:3, Insightful)
"I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!"
Quite to the point. There's a rising trend on questions like "I'm having problems about X but I prefer being cheap on coping with X, what should I do?". And then there's the rising trend on answering something else than "if you depend on X you'll need to invest on X".
And now, about the problem at hand: your (his) logs attest thousands of *failed* connection attempts through ssh. In the end... so what? Failed attempts are neither worrying nor in your hand to be avoided (changing your ssh port is the most cost effective remediation measure to maintain bots at a distance). Bellovin already stated quite some years ago (not his exact words, but their meaning): do not monitor on the outer side of your firewall if you are not in the position of avoiding the attacks; all you get will be headaches and a ton of meaningless logs to parse (and probably losing important information among the noise). The verily reason for your firewall to be there is already to maintain the attacks as failed attempts anyway. Do monitor from the inner side. In this case, monitor successful logins, not failed ones.
Re:Move to a higher order port and use denyhosts (Score:3, Insightful)