Coping With 1 Million SSH Authentication Failures? 497
An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"
fail2ban (Score:5, Informative)
fail2ban, key-auth
Ignore it? (Score:1, Informative)
If you have a public site, it'll get scanned night and day by APNIC. If they see a banner for SSH, they'll jump on that. As long as you are using patched software and have good user security, what's the real emergency?
Hardware firewall or use bfd (Score:3, Informative)
Tar Pitting (Score:5, Informative)
I'm a big fan of tarpitting SSH connections myself. It dramatically cuts down on the repeated ssh attempts, and keeps my logs much cleaner.
Basically, an iptables rule:
-A INPUTCHAIN -m state --state NEW -m recent -p tcp --dport 22 --update --seconds 30 --hitcount 4 --rttl --name SSH -j DROP
Re:fail2ban (Score:5, Informative)
> fail2ban, key-auth
+1
Change port. Use iptables to only allow access from known subnets/hosts.
Re:Tar Pitting (Score:2, Informative)
Fail2ban [fail2ban.org] can do stuff that automatically!
Passwords? (Score:5, Informative)
Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.
fwknop (Score:5, Informative)
You are being brute-forced (Score:4, Informative)
Welcome to the internet -- this happens to every site with a public IP.
First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.
There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.
The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.
Fail2ban or denyhosts (Score:4, Informative)
SSH public key authentication (Score:4, Informative)
Spend more than 10 minutes? (Score:3, Informative)
Seriously. I'm in the same shoes and it's easy. I came across it pretty quick and all these SSH log in problems went away.
Check out...waaaaait...
What's most troubling is this:
I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses
You operate a business and you haven't figure out this chief problem yet...but you want us to help you out?
Well...Here's the solution: denyhosts [sourceforge.net]
Someone else'll chime in with it....Just hope you read up and configure it properly....or you'll find yourself locked out of your own servers.
Move to a higher order port and use denyhosts (Score:4, Informative)
I use one or more of these on my public facing servers.
1. Move the default ssh port to a higher order port (5000+)
2. Use Denyhosts http://denyhosts.sourceforge.net/ [sourceforge.net] to block repeated attempts
3. use key exchange instead of username/password
4. use network based IPS.
Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.
Re:Tar Pitting (Score:4, Informative)
BlockHosts (Score:3, Informative)
We started using BlockHosts to feed iptables rules, and our failure logs went from 30-50k per day to 100. Basically, with more than 'x' failed logins within 'y' time frame, the source IP is blocked for 'z' time period. Since it uses iptables, you could block it from just the ssh port, or the entire system (we do the latter).
All three variables are configurable, and we also have whitelisted a few select standby IPs for contingency use. (As another poster said, you **will** lock yourself out eventually.)
Throttle Inbound Connections (Score:5, Informative)
I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.
1. Limit incoming SSH attempts to a low number. In my case I limit to 2 connections in 60 seconds. I can tighten it even more but this did a lot to kill brute force attempts.
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -j DROP
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state NEW -m limit --limit 2/min -j ACCEPT
iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state RELATED,ESTABLISHED -j ACCEPT
2. Automatic blacklist via DenyHosts. This helps cut down attempts from known ranges without even giving them the chance even at a slow rate. http://denyhosts.sourceforge.net/
Re:Spend more than 10 minutes? (Score:5, Informative)
Re:fail2ban (Score:2, Informative)
Install OSSEC, too. There may be other stuff going on.
Re:fail2ban (Score:3, Informative)
And use DenyHosts [sourceforge.net]
hashlimit (Score:5, Informative)
You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.
You might check out the end of this presentation that I made 4 years ago.
http://www.bloomingtonlinux.org/wiki/15th_meeting [bloomingtonlinux.org]
I've been using hashlimit successfully for years in a web hosting environment.
I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!
Re:Tar Pitting (Score:5, Informative)
Any site that connects more than 4 times in 60 seconds gets all packets subsequently dropped.
You could change DROP to TARPIT (if your kernel supports it) to fuck with them a little more.
Re:fail2ban (Score:5, Informative)
Re:fail2ban (Score:2, Informative)
Or block when connections per time interval exceed a given number:
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
In this case, more than 4 attempts in a minute
Re:fail2ban (Score:2, Informative)
This is really cool until you find yourself trying to log in from the same access point where somebody with a virus was attached earlier in the day. Better to just use crypto (key-based authentication only) and rate-limiting.
That depends on how you have Fail2Ban set up. If you have it set to block for a very long period of time, then yes, this would be a problem. But I have it set to just block for 10 minutes. Even if I manage to lock myself out accidentally, I only have to wait 10 minutes. That short 10 minute block is usually more than enough to make bots move on. I've had a few of them continue to hammer away after the 10 minutes are up, but in a couple years it's been a very small percentage.
And to the guy who suggested using DenyHosts in addition to Fail2Ban, you're wasting your time. Fail2Ban performs the exact same thing as DenyHosts, but Fail2Ban does it for more services besides SSH. They're both good tools, but using them together is just unnecessary.
Re:fail2ban (Score:5, Informative)
fail2ban firewalls off the port for a time you specify
DenyHosts blocks the ip in /etc/hosts.deny
I find fail2ban to be much more effective since I can use it for more than just SSH (on my system: ftp, imap, pop3, ssh, smtp). Some of the newer botnets will attempt to crack the password using another service and then try the resulting password on ssh so it's important to have more complete coverage.
proactive blocking (Score:4, Informative)
In addition to using reactive tools like fail2ban and denyhost, also block most of the world proactively:
git clone git://github.com/bugi/iptables-by-country.git
SSH Key (Score:3, Informative)
Instead of generating passwords, have your users supply their SSH key and turn off passwords. Much better solution.
Re:good against 1 attacking host -- not large botn (Score:2, Informative)
- you configure several wrong password attempts and/or shorter blocking times - then the attacks do not get slowed quite as much, but it hinders your users less.
There are some things you might consider, though -
3 fails / 10 min != 10 fails/ 1 sec
Re:fail2ban (Score:3, Informative)
More accurately it blocks the attacker's IP at the firewall.