Forgot your password?
typodupeerror
Security

Coping With 1 Million SSH Authentication Failures? 497

Posted by kdawson
from the some-definitions-of-managed dept.
An anonymous reader writes "I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses. Our production servers, which host about 50 sites and generate ~20K hits/week, are managed by a 3rd party that I'm sure many on Slashdot would recognize. Earlier today I was researching some problems on one of our sites and found that there have been over 1 million SSH authentication failures from ~1200 IP addresses on one of our servers over the last year. I contacted the ISP, who had promised me that server security would be actively managed, and their recommendation was, 'change the SSH port!' Of course this makes sense and may help to an extent, but it still doesn't solve the problem I'm facing: how do you manage server security on a tight budget with literally no system admin (except for me and I know I'm a n00b)? User passwords are randomly generated, we use a non-standard SSH port, and do not use any unencrypted services such as FTP. Is there a server monitoring program you would recommend? Is there an ISP or Web-based service that specializes in this?"
This discussion has been archived. No new comments can be posted.

Coping With 1 Million SSH Authentication Failures?

Comments Filter:
  • fail2ban (Score:5, Informative)

    by Anonymous Coward on Saturday March 06, 2010 @08:54PM (#31385092)

    fail2ban, key-auth

    • Re:fail2ban (Score:5, Informative)

      by jimpop (27817) * on Saturday March 06, 2010 @08:57PM (#31385130) Homepage Journal

      > fail2ban, key-auth

      +1

      Change port. Use iptables to only allow access from known subnets/hosts.

    • Re: (Score:2, Informative)

      by sstern (56589)

      Install OSSEC, too. There may be other stuff going on.

    • Re:fail2ban (Score:5, Funny)

      by Kugrian (886993) on Saturday March 06, 2010 @10:18PM (#31385724) Homepage

      Someone fucked up big time when they taught you html:
      [url]http://www.fail2ban.org/[/url]

    • hashlimit (Score:5, Informative)

      by suso (153703) * on Saturday March 06, 2010 @10:32PM (#31385796) Homepage Journal

      You can also use the hashlimit module for iptables. I find it works pretty well for allowing people in who need to access and block things like the ssh worm that was causing ssh to run out of resources.

      You might check out the end of this presentation that I made 4 years ago.

      http://www.bloomingtonlinux.org/wiki/15th_meeting [bloomingtonlinux.org]

      I've been using hashlimit successfully for years in a web hosting environment.

      I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!

      • Re: (Score:3, Insightful)

        by JWSmythe (446288)

        I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator.

        That's probably the most relevant thing I've seen. I'm amazed at how many places don't want or have people with a clue working on their equipment. TFA said they have *NO* sys/net admin, and the guy who attends to those task is a "n00b".

        There's more than one way to address th

      • Re: (Score:3, Insightful)

        by turbidostato (878842)

        "I'd also recommend hiring someone who focuses their efforts on managing the servers and taking care of network security. This job is usually referred to as a system administrator or network administrator. DOH!"

        Quite to the point. There's a rising trend on questions like "I'm having problems about X but I prefer being cheap on coping with X, what should I do?". And then there's the rising trend on answering something else than "if you depend on X you'll need to invest on X".

        And now, about the problem at h

    • Re: (Score:3, Interesting)

      by DeadboltX (751907)
      is there any advantage to fail2ban over denyhosts ?
    • I am using fail2ban, but I do not think it's a particularly great tool when you see how many different IP addresses the attacks come from, so you're somewhere stuck in the middle trying to optimise how to make fail2ban more effective, while not being a problem for your own users:

      - you configure blocking from 1 login attempt and a long block time cuts down on most of the outside attacks, but if some legitimate user mistypes his password, he'll be locked out for that same long block time...
      - you configure sev

  • by Golbez81 (1582163) on Saturday March 06, 2010 @08:55PM (#31385106)
    bfd (and apf if you like it) are probably the best software solutions your gonna find, but if you're facing 1 million+ auth failures, I would seriously consider a hardware firewall and VPN of some sort.
  • Tar Pitting (Score:5, Informative)

    by spydum (828400) on Saturday March 06, 2010 @08:55PM (#31385108)

    I'm a big fan of tarpitting SSH connections myself. It dramatically cuts down on the repeated ssh attempts, and keeps my logs much cleaner.

    Basically, an iptables rule:
    -A INPUTCHAIN -m state --state NEW -m recent -p tcp --dport 22 --update --seconds 30 --hitcount 4 --rttl --name SSH -j DROP

    • Re: (Score:2, Informative)

      by Anonymous Coward

      Fail2ban [fail2ban.org] can do stuff that automatically!

      • Re:Tar Pitting (Score:5, Insightful)

        by Minwee (522556) <dcr@neverwhen.org> on Saturday March 06, 2010 @09:30PM (#31385434) Homepage

        Fail2ban can do stuff that automatically!

        Let me see if I have this right. If iptables can keep count of incoming connections within the kernel and drop incoming connections as they happen without any intervention from a userspace program, that's _not_ automatic.

        But running a gigantic shell script to scrape text messages out of your system logs and then call randomly chosen commands which may or may not have any effect on the observed problem some time after it occurs, then that's "doing that stuff automatically"?

        Maybe those words don't mean what one of us thinks they do.

    • Re:Tar Pitting (Score:4, Informative)

      by jonesy2k (934862) on Saturday March 06, 2010 @09:11PM (#31385254) Homepage
      That's not tarpitting. Tarpitting involves keeping the connection open in order to tie up the resources of the attacking host.
    • Re:Tar Pitting (Score:5, Interesting)

      by Anonymous Coward on Saturday March 06, 2010 @09:14PM (#31385284)

      I use a variation on tarpitting that has worked very well for me.
      It cut the attempts down from 60,000/day to 20 or 30 per day.

      I just add a small delay between the initial connection attempt
      and when I send the username/password prompt. The delay (in
      seconds) is the number of attempts in the last 30 minutes,
      squared. This makes all but the most determined attacker
      give up and go away very quickly.

      I have been using this with both FTP and SSH for the last
      year, and it works great.

    • Re: (Score:2, Insightful)

      by luizd (716122)
      Yes, I think that this is your best solution. Instead of using iptables directly, you should check your distribution software relative option.Generally is is easier (but anyway, it will ultimately result in the same iptables command)
    • Re:Tar Pitting (Score:5, Insightful)

      by cptdondo (59460) on Saturday March 06, 2010 @09:35PM (#31385484) Journal

      That, and create only a single user who can log in, that takes you to the real log in prompt. That way an attacker has to guess the one user+password, and have a legitimate userid+password to gain access.

      It's not foolproof, but it foils the vast majority of script attackers.

      And DISABLE ROOT LOGIN!

  • sshblack (Score:2, Insightful)

    by Anonymous Coward

    sshblack works very well for me. It monitors failed ssh connections and inserts entries into iptables to block the IPs.

  • if you can manage the set of users of your server, you can use OpenVPN and then SSH. OpenVPN has a "feature" that if each packet of the VPN is not digitally signed by a previously arranged (and distribuyed) key, then the packet is "ignored". After the VPN session is established/authenticated, your users can log in using ssh. There are even some virtual appliances and special distros (untangle.com) that have a "openvpn appliance" built in for this purpose. The how-to for openvpn is also easy to follow.
  • Passwords? (Score:5, Informative)

    by fm6 (162816) on Saturday March 06, 2010 @08:58PM (#31385142) Homepage Journal

    Here's my excuse to ride my usual hobbyhorse about passwords being obsolete. SSH supports certificate-based authentication, which is not only more secure, it's less of a hassle for the user. Don't know if it would be practical for your application (you might tell us more about that) but it's worth a look.

    • by timmarhy (659436)
      certs are harder for a user to setup, which means support time which cuts into a small time operation a lot.
      • I agree with both parent posts. PKI Certs are certainly the way to go, but it's really hard to do this right.

        This is a case where some consulting to (a) set up the PKI stuff; (b) train our (unfortunately anonymous) questioner on how to disseminate the certs; and (c) apply the appropriate tarpit/other firewall settings, would probably be money-well-spent.

    • Re:Passwords? (Score:4, Interesting)

      by markdavis (642305) on Saturday March 06, 2010 @10:24PM (#31385754)
      I agree that PKI would be more secure, but it is a LOT more hassle for most users. The simple fact is, a tarpit is *extremely* effective. Even a relatively weak password is going to be nearly impossible to break if the attackers are limited to only a few tries per hour or day or whatever.
  • fwknop (Score:5, Informative)

    by c0d3g33k (102699) on Saturday March 06, 2010 @08:58PM (#31385144)
    They can't fail authentication if they don't know it's there. http://cipherdyne.org/fwknop/ [cipherdyne.org]
    • by mprinkey (1434)

      I agree completely. We use fwknop as a part of a required two-factor authorization scheme for a US Government customer. It is a minor inconvenience to use it to open the ssh port before connecting, but it virtually eliminates attack attempts. How can you hack a closed port?

  • by Anonymous Coward on Saturday March 06, 2010 @08:59PM (#31385150)

    Welcome to the internet -- this happens to every site with a public IP.

    First off, do not change your SSH port. It won't do a whole lot for you, and it will be more hassle than it works.

    There are a whole lot of programs available to deal with SSH brute forcing. sshguard is one of them, and it's not too bad (you can apt-get install it). It's a bit of a hack -- it just watches your logs and takes appropriate action -- but it does work.

    The other thing you can do immediately is simply turn off password authentication in ssh. Anyone getting in will need a key. This is what sourceforge and github both do. This isn't always practical for every site, but it will damn sure keep your passwords from being brute-forced.

  • Use public key authentication. It's super easy and much more secure, unless your random passwords happen to be over 300 characters long (ssh-keygen makes 2048-bit keys by default; assume 6 bits per character in a random password if each character is in the base64 set). Also, sshguard is your friend. After a small number of invalid logins - 4 by default - that IP gets firewalled for a short period of time.

  • by mystik (38627) on Saturday March 06, 2010 @08:59PM (#31385158) Homepage Journal
    Fail2ban [fail2ban.org] or denyhosts [sourceforge.net] activly target ssh. fail2ban includes rules for other services, but denyhosts includes a mechanism for sharing lists of your denied hosts w/ other admins, as well as using their ban lists to protect your ssh logins.
  • by whamett (917546) on Saturday March 06, 2010 @08:59PM (#31385166)
    This kind of brute-forcing is common. The simplest way to deal with it is to set up SSH public key authentication, disable SSH password authentication, and forget about it.
  • If your not willing to invest money into intrusion detection, vulnerability scanning, or moving to a more responsive provider, accept that your server is going to constantly be scanned by the droves of script kiddies out there throwing everything but the kitchen sink at any server that responds to a ping and keep monitoring those logs, which hopefully are stored on a separate server in case you are ever actually compromised. In the mean time, try installing Fail2Ban on your server. It will block an IP addre

  • by Seakip18 (1106315) on Saturday March 06, 2010 @09:02PM (#31385186) Journal

    Seriously. I'm in the same shoes and it's easy. I came across it pretty quick and all these SSH log in problems went away.

    Check out...waaaaait...

    What's most troubling is this:

    I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla for small businesses

    You operate a business and you haven't figure out this chief problem yet...but you want us to help you out?

    Well...Here's the solution: denyhosts [sourceforge.net]

    Someone else'll chime in with it....Just hope you read up and configure it properly....or you'll find yourself locked out of your own servers.

    • by Gearoid_Murphy (976819) on Saturday March 06, 2010 @09:21PM (#31385338)
      add the ip address and/or hostname of all the hosts you use to access your servers into /etc/hosts.allow. If denyhosts picks up 3 failed logins from a single ip address, that address is added to /etc/hosts.deny, if this happens to be your machine (and you're having a bad day entering your password), you could get locked out of your system.
  • Great tool that pools the resources of multiple servers to proactively block IPs that are trying to brute-force the SSH port on any server in the pool:

    http://denyhosts.sourceforge.net/ [sourceforge.net]

    I use it, and it has Just Worked for years.

  • So? (Score:5, Interesting)

    by victim (30647) on Saturday March 06, 2010 @09:04PM (#31385204)

    If you really have secure passwords, the random guessers aren't going to get them. Log it and move on.

    I get thousands of Chinese hackers attempting to break into the battery monitor in my tool shed, big deal. I don't know why my battery voltage and solar current is so important to them, but they can knock themselves out all day.

    If the logs bother you then install fail2ban and configure it to lock people out after a few bad guesses. (Then be ready to unlock yourself from an alternate IP when you inevitably lock yourself out.)

    • Re: (Score:3, Interesting)

      by Korin43 (881732)
      I'm surprised I had to scroll down half the page to find this. Seriously, what is everyone so worried about? Just set a secure password and don't even worry about it. Assuming you have a secure password (8 characters, upper and lower case letters and numbers), even a million attempts per second will take over 3 years on average [google.com] (and ssh won't let you try that often anyway). Bump that up to 12 characters and you're looking at millions of years [google.com].
  • by deadmongrel (621467) * <karthik@poobal.net> on Saturday March 06, 2010 @09:07PM (#31385214) Homepage

    I use one or more of these on my public facing servers.
    1. Move the default ssh port to a higher order port (5000+)
    2. Use Denyhosts http://denyhosts.sourceforge.net/ [sourceforge.net] to block repeated attempts
    3. use key exchange instead of username/password
    4. use network based IPS.
    Just moving the ssh port reduced the ssh brute force attack for me. Either stop being a noob or hire a sys admin.

    • by sootman (158191) on Saturday March 06, 2010 @09:12PM (#31385258) Homepage Journal

      1. Move the default ssh port to a higher order port (5000+)

      Agreed. The higher the better. For the ultimate in security, I recommend 65536.

    • Re: (Score:3, Interesting)

      by MrCrassic (994046)

      Changing the port might be troublesome if you're trying to SSH from public locations like airports and such. They usually leave port 22, 443 and 8080 open for their own services and block all others, so you will have to either set up VPN to get into your servers or wait until you're elsewhere. Additionally, an evenly-slightly determined hacker can still find where your sshd process is listening on by running a basic nmap scan.

      Key exchange and disabling root login is probably the best way to go. Fail2ban wor

  • BlockHosts (Score:3, Informative)

    by psychosis (2579) on Saturday March 06, 2010 @09:12PM (#31385266)

    We started using BlockHosts to feed iptables rules, and our failure logs went from 30-50k per day to 100. Basically, with more than 'x' failed logins within 'y' time frame, the source IP is blocked for 'z' time period. Since it uses iptables, you could block it from just the ssh port, or the entire system (we do the latter).
    All three variables are configurable, and we also have whitelisted a few select standby IPs for contingency use. (As another poster said, you **will** lock yourself out eventually.)

  • by Padrino121 (320846) on Saturday March 06, 2010 @09:15PM (#31385288)

    I have a similar situation and cannot limit to very specific IP ranges. I have done the following with good success. I pulled some examples from my configuration that can be tweaked for yours if you like.

    1. Limit incoming SSH attempts to a low number. In my case I limit to 2 connections in 60 seconds. I can tighten it even more but this did a lot to kill brute force attempts.
    iptables -I INPUT -p tcp -i vlan1 --dport 2242 -j DROP
    iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state NEW -m limit --limit 2/min -j ACCEPT
    iptables -I INPUT -p tcp -i vlan1 --dport 2242 -m state --state RELATED,ESTABLISHED -j ACCEPT

    2. Automatic blacklist via DenyHosts. This helps cut down attempts from known ranges without even giving them the chance even at a slow rate. http://denyhosts.sourceforge.net/

  • by Simon Carr (1788) <slashdot.org@simoncarr.com> on Saturday March 06, 2010 @09:15PM (#31385292) Homepage

    -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource -j ACCEPT
    -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j LOG --log-prefix "SSH_brute_force "
    -A INPUT -p tcp -m tcp --dport 22 -m recent --update --seconds 60 --hitcount 8 --rttl --name SSH --rsource -j DROP

    Stops 'em *somewhat* dead. If you want to whitelist hosts so they are not impacted by this rule, just add;

    -A INPUT -s your.ip.address -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT

    Before the throttling ruleset.

  • http://www.fail2ban.org/wiki/index.php/Main_Page [fail2ban.org]

    That should cover automatic banning of repeat offenders. It works by reading logs, so it can work with many services. You could for example make it block all those attempting to exploit your CMSes.

  • OpenBSD PF (Score:4, Insightful)

    by roman_mir (125474) on Saturday March 06, 2010 @09:16PM (#31385298) Homepage Journal

    #SSH
    pass in inet proto tcp from any to $ext_if port ssh
    pass out inet proto tcp from any to $ext_if port ssh
    pass quick proto tcp from any to $ext_if port ssh \
            flags S/SA keep state \
            (max-src-conn 15, max-src-conn-rate 5/3, \
              overload <bruteforce> flush global)

    you know what I am saying?

  • pam_abl (Score:3, Interesting)

    by otis wildflower (4889) on Saturday March 06, 2010 @09:17PM (#31385312) Homepage

    http://lmgtfy.com/?q=pam_abl [lmgtfy.com]

    Share and enjoy!

  • by dingen (958134) on Saturday March 06, 2010 @09:21PM (#31385340)

    So there have been a million failed attempts to log in over SSH in a year. That's about one failed attempt every 30 seconds.

    What's wrong with that? It's not like they're getting in, right?

  • Peter N. M. Hansteen has written a nice article about a similar atack. http://bsdly.blogspot.com/2009/11/rickrolled-get-ready-for-hail-mary.html [blogspot.com] The first thing I would do (at install time) is to disable root login over ssh.
  • OSSEC [ossec.net] is what I've been using for years and it works well. IT's much more comprehensive a security package than fail2ban but uses log monitoring as it's basis.

    It's worth a look.

  • by Beached (52204)

    Change the port and use private certificates for authentication.

    However, changing the port to something will get the bots off your back. They still don't try other ports.

  • Although the solutions like fail2ban and denyhosts are undeniably useful, you will cut down well over 90% of the drive-by ssh attempts by simply changing the port.

    None of the scripts, that I'm aware of, actually portscan your machine looking for the SSH login headers, they all hit port 22 and try to log in. It takes too long to scan a machine so if nothing is listening on port 22, they will move on.

    You will still get the odd attempt from someone who has portscanned you - these will be relatively rare. In th

    • Re: (Score:3, Interesting)

      by Abcd1234 (188840)

      I actually go one step further: after IPv6-enabling my site, I only allow v6 ssh inbound. Since Teredo makes it possible to get v6 from nearly anywhere, it doesn't cause any inconvenience, and ssh attacks have basically vanished. 'course, it won't last forever, but it works great right now.

  • RSA keys are notoriously time intensive to brute-force. As far as I know, no one is crazy enough to automate an attack. If you disable password authentication altogether, you eliminate your risk of an automated dictionary attack. It might confuse the hell out of some users, which is a con.
    • Protocol 2
    • Set PermitRootLogin to "no" or "without-password"
    • ChallengeResponseAuthentication no
    • PasswordAuthentication no

    And then just ignore the attempts. fail2ban can sometimes be ok, but be aware that it creates a denial-of-service vulnerability that is exploitable by attackers who can can spoof source addresses.

    • fail2ban can sometimes be ok, but be aware that it creates a denial-of-service vulnerability that is exploitable by attackers who can can spoof source addresses.

      Yeah my server went off line a week ago and while debugging the problem I noticed the kernel complaining at startup that a file included from my pf configuration was corrupt. Its a script I wrote which updates that file, not fail2ban. The file was my blocked hosts table.

      So from now on I am just going to put up with the dictionary attacks.

  • I hate to sound unsympathetic here, but the Internet is a hostile place. If you have port 22 open to the world, you should expect to get pummelled with password cracking attempts more or less constantly.

    Either learn to live with it, or at least take some steps to slow them down. I find that throttling back the number of connections any one IP address can make in a short time pretty much slows it down to a reasonable level. Alternatively, you can also put some "trap" logins on your system. Usernames lik
  • If you are randomly generating your passwords and they are of a decent length then you don't really need to do anything. If your passwords contain lower-case letters only (not recommended), but are eight characters long then your million authentication attempts would represent only a 0.0005% chance of success. If you passwords contain numbers and upper-case characters too, then the likelihood is 1000 times less.

  • I own a small Web development studio that specializes in open source software, primarily Drupal, WordPress, and Joomla

    Right off the bat, you're far too focused on the security of least concern, ssh. Unless you set an idiotic password, it's not going to be guessed. It sounds like you have that covered.

    But.. the part you've missed is your FAR more vulnerable applications. You say your ISP is "actively managing" your security, but how certain are you they're doing a decent job of it? Do they have experts

  • Somebody might want to check my math here...my stats are a bit rusty

    Ok let's assume a few things:
    1) your passwords are 8 chars long and use upper and lowercase alpha and numbers only
    2) they are essentially random passwords that are NOT in the dictionary.

    This would mean that there are 2^62 possible passwords in your password space. That's 4,611,686,018,427,387,904 possible passwords. The chances of guessing one password in that space within a million random tries is 1,000,000/4,611,686,018,427,387,904. Tha

  • That should keep the CPU usage down, and your system secure from that type of attack. You can’t do much more than that.

    You could run a script to find the owner of all those IPs... maybe do a bit of statistics to see what you can find out... But in the end it will be someone that you can not sue or contact in any way anyway.

  • SSH Key (Score:3, Informative)

    by rawg (23000) <phill&kenoyer,com> on Sunday March 07, 2010 @03:14AM (#31387302) Homepage

    Instead of generating passwords, have your users supply their SSH key and turn off passwords. Much better solution.

  • Lock it down. (Score:3, Insightful)

    by BlackHawk-666 (560896) <ivan.hawkes@gmail.com> on Sunday March 07, 2010 @04:57AM (#31387706) Homepage

    I can't believe the person who set the server up left this port open to the world. If you want the server secure then you will close all the ports that are not actually needed - in your case this likely means you keep 80 and 443 open.

    For remote management make get a few IPs for the desktops that are meant to connect to it and make sure the ISP opens SSH port only to the IPs.

    You can tunnel X over SSH and do file management, whatever you need so it should be enough.

    Don't bother changing it to some random port, security through obscurity is total bullshit in this age of port scanners.

"If value corrupts then absolute value corrupts absolutely."

Working...