How To Prevent Being Hacked Via Backups? 214
Popsikle writes "A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"
Easy fix (Score:5, Insightful)
Offline and offsite storage (i.e. iron mountain) is a simple (though sometimes costly) way of doing things.
it'll solve this problem quite easily.
Tachikoma (Score:5, Insightful)
Take a lesson from Ghost in the Shell, hire digital Tachikoma to protect you :) Problem solved!
Encrypt it (Score:5, Insightful)
Encrypt your backups.
Don't let your backup system have access to your main system.
Allow your main system write-only access to your backup system, for the sole purpose of delivering new backups.
Re:Encrypt it (Score:3, Insightful)
Ammendum:
Anything else needing to be done would require a admin to do it, and that's the point.
Why were your backup servers (Score:5, Insightful)
Isolate the Backup Servers (Score:2, Insightful)
Why were your backup servers accessible from the outside network? Your backup servers are arguably even more valuable than your production servers. They should be behind yet another firewall. You can even have the production servers connect to them through a separate network interface. (Network interfaces and Cat5 cables are cheap.) If you are really paranoid, you can create folders where the server can upload data, but can't erase or overwrite what it has uploaded.
Why was this possible? (Score:4, Insightful)
I don't understand. The attackers gained access to the database...through the backup servers? That leaves me with two questions:
1. Why were the backup servers accessible to the attacker?
2. Why was the database accessible from the backup servers?
It seems to me that the only way you need to access the backup servers is through some mechanism that allows you to transfer data to and from them. A single open port, which you need a password (or key) to use seems all that should be exposed. That shouldn't be too hard to secure.
It also seems to me that the backup server has no business accessing the database, and therefore shouldn't be able to access the database.
Unless, of course, the way the system works is that the backup server accesses the production server to retrieve the data from it. That doesn't seem the most obvious design to me, but it would at least explain why the backup server could access the database. Maybe that is a good reason not to design the system that way (on the other hand, it saves complexity on the production server, which is good). At any rate, it doesn't answer the first question, which is why the attackers were able to access the backup server.
My sympathy goes out to the WHT administrators. Good luck on recovering from this and figuring out what went wrong. I hope you will keep us posted, so that we can all learn from this incident.
Backups (Score:3, Insightful)
Tape drive (or even other external device) with encrypted data. There, problem solved. Do you see why such devices/features are standard on anything that has the word "backup" on it?
However, if you insist on having servers running all day long that you want to backup to, then at least make sure they are a million times more secure than your production servers - as the name suggests, they are your BACKUP if anything goes wrong. This means - encrypted data, locked-down networking (i.e. absolute minimum necessary - one port, one user, one ultra-secure key), proper permissioning (it might not be a bad idea to set the permissions so that you can append to a file but can't read it, thus forcing you to have physical access in order to restore any data from backup, or certainly that you can't overwrite existing files) and physical security (i.e. properly hosted in a decent hosting outfit and/or securely placed in an offsite location where they can't be got at - even if this is in a secure cage in a branch office).
And letting the backup server have ANY permissions on your local network is just stupid. Push, don't pull. Tell the backup server what to backup, don't let it pick and choose and require access back to your network - it's a backup device not "just another machine". A simple "nothing outbound" firewall rule on that machine would have stopped them coming back to you, providing the backup was encrypted (I'm assuming they actually piggy-backed onto your network than stole the db passwords from your backups). And the backup should ALWAYS be encrypted because it might well contain information such as your passwords in it, especially so if you are storing other people's data.
Yeah, it costs money to do this properly, but that's the price you pay for not losing thousands of people's data. I imagine the kick-back from that data loss will run to more than the price of a tape drive or two in the long run. What they had was NOT a backup. It was a rapid-restore machine. That's fine to have *as well as* a backup, but no better than hanging a 250Gb USB drive off the database server (in fact, worse, because that machine was able to be remotely-compromised).
Re:Why was this possible? (Score:2, Insightful)
Unless, of course, the way the system works is that the backup server accesses the production server to retrieve the data from it. That doesn't seem the most obvious design to me, but it would at least explain why the backup server could access the database. Maybe that is a good reason not to design the system that way
That is a rather common design in backup systems, and there are good reasons for it: mainly, it allows securing the backup server better, especially in the (common) situation where one backup server handles several production machines.
If the backup servers offer only read-only access to anyone over the net (or not even that if recovery from backup is rare enough to be initiated from backup server console), it can be secured much easier than when if offers write access to clients backing themselves up.
[...] it doesn't answer the first question, which is why the attackers were able to access the backup server.
You are right there, though.
Re:Easy fix (Score:5, Insightful)
HDs are NOT backup media.
Please provide a citation that hard disks are noticeably worse than tape, which you appear to otherwise recommend.
Re:Easy fix (Score:5, Insightful)
No, they are not, you are correct. In my post, I was assuming that this was a small website or business, not a mission critical company product. I didn't mention software, or tape libraries, or hot backups. I think sometimes its better to have some backups, then none at all. An external drive can be bought for next to nothing.. really, I can get a 500GB HD for about the cost of a couple of tapes, but then I have to buy 2 tape drives (in case one has hardware failure).
Believe me, I could go on about backup windows, media, and techniques, but was hoping by keeping it simple, they would not be overwhelmed. By not being overwhelmed, they might start the process.
Also, by not using a computer based backup, they would not have the same problem as the site mentioned in the linked article, where someone first cracked their backup servers, and deleted their only backups.
What do encrypted backups solve? (Score:2, Insightful)
A lot of people are suggestion that backups be encrypted and I assume they mean the actual files/volumes, but I *really* don't see the usefulness of this case. Encryption backups may protect the data from being stolen, but it isn't going to protect it from being *wiped* in most cases. If you have root access then assumedly there's also raw access to device the backups reside on, in which case you can still nuke it with something as simple as "dd if=/dev/zero of=/dev/backup"
Now maybe if the backup server further connects onward to a mass-storage device that isn't at-all accessible until logged-in, perhaps, but it doesn't seem a likely scenario for most cases.
Re:Encrypt it (Score:3, Insightful)
Don't let your backup system have access to your main system.
Ah, but this is a large company, and presumably they use 'enterprise' products. Some (most?) of those tend to be secured with rexec equivalent security (nsrexecd and .nsrhosts, coincidence eh?) for jobs executed from the backup server, and any better security is an afterthought. But they have good salesmen, and they're 'enterprise', and they cost a lot of money so they must be safe to use right...
The sad state of affairs is that at many large corporations it's probably as easy as sticking a laptop into the right network port, configuring the same IP address as the backup servers and you've got root access on every machine they back up, either through script facilities or through the simple capability of pushing a restore of a (completely unchecked) home-made passwd or any other file.
Perhaps it's changed in the last few years, but last I checked there were no host keys to authenticate servers and clients, there were at best simple passwords for the server to make the client do whatever it wanted and there were no checksums to verify that only previously backed up files got pushed to the client. I may also have missed some more capable system.
Then again, many large companies separate responsibility for backup and storage systems from system administration, and/or the people deciding what backup system to use have little experience with system security. Leading to exactly this kind of problem, with huge gaping security holes. Monitoring software is another one of those huge holes.
Preventing being hacked from the backup system is, of course, fairly trivial. The first part of which would be, don't use any backup system that creates huge security holes on your servers, or at least don't put the security hole, or 'backup client' on the servers.
For the second part I'm leaning towards multi-stage backups, particularly if you have separate areas of responsibility; have the server admins being responsible for delivering production data securely to an intermediate storage area (perhaps coalesce the data into a single compressed encrypted, signed file), then let the people responsible for backups dump it to tape/archive it.
It creates a whole other bunch of problems and potential issues of course (and solves another bunch of issues with backup clients that don't understand all file types, file systems, metadata, etc), and you may have to take a serious look at database backups and such, but in the choice between bad and some more bad, at least such a solution won't lube you up for a penetration, it'll be fairly cheap (licenses vs. diskspace), it gives you much flexibility on how to backup which data, and it should keep your data safe. It may cost more man hours on the system side, but it'll probably save them on the storage side with fewer backup clients. The joy the first time you get to do a several-minute restore instead of getting an 'oops-saved-the-data-on-every-tape-in-the-silo' experience waiting a day for less than a hundred gigabytes of small files getting restored will just be another bonus.
Misdirected frothing. (Score:3, Insightful)
HDs are NOT backup media.
It appears you've taken the oft-repeated mantra that RAID is not a backup solution and gone a step further by suggesting that hard drives themselves aren't backup media.
Which, by the way, is ludicrous. Hard drives use tried-and-true technology, they're cheap as hell, and transfer speeds are outrageously faster than any other media in contention. Suggesting that they don't make good backup media is well, I'll give you the benefit of the doubt and say misguided.
Tapes Are Rubbish (Score:3, Insightful)
yes, because people regularly break into my NOC, pull the drives out of my backup servers and practice juggling. Tapes are rubbish. Tape is expensive and unreliable. Anyone that tells you otherwise is selling the stuff.
Hard drives have replaced tapes around here. For decades we juggled tapes in a vain attempt to maintain useful backups only to find that EVERY SINGLE TIME we needed critical data from backup, the data was unrecoverable.
For the past few years now, we have been backing up all of our critical data to a low powered server with TB drives in mirrored arrays. Security on this server is exteremly high. The only service it runs is SSH, backups are all done as pulls from the servers, has no untrusted local users and sits on an extremely restriced network segment.
With the thousands we have saved not replacing tape media and drives (not to mention the amount of overtime not wasted screwing around in the middle of the night trying to find a working tape to restore from) we are adding a mirror for this archive offsite.
Re:truncate immediately visible post length (Score:3, Insightful)
i dont want thresholds. i want to be able to see valid comments even if some fanboi mods someone's valid, insightful comment -1 due to fanboi spirit.
Re:Tapes Are Rubbish (Score:2, Insightful)
Expensive, sure. But they most definately are reliable. Your experience may have been tainted by cheap PC crap intended for stupid home users -- QIC-80 anyone? Modern DAT, AIT, DLT, and LTO tapes are infinately more reliable than a hard drive. I have aging Exabyte 8mm tapes that are over 20 years old and still perfectly readable. (if you can find a drive old enough to read 'em.) The only drives I have approaching that age that are still functional are highly expensive SCSI drives.