How To Prevent Being Hacked Via Backups? 214
Popsikle writes "A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"
eggs in multiple baskets (Score:5, Interesting)
1) divide your eggs in at least two baskets, thoughtfully designed to protect their integrity
2) keep your baskets in physically isolated locations
3) take steps to protect your eggs from theft
4) after retrieving your eggs, inspect them for tampering before using them in your souffle
5) purchase insurance for the off chance you get yolk on your face
Re:Easy fix (Score:5, Interesting)
Even easier.. put a external HD with Truecrypt under your bed. Maybe take a second one into the office, Lock it in a drawer, just in case your house burns down..
Every week or so, use 1 to backup. Alternate which one. Add more drives, or more often backups until you get to the point you sleep easy at night.
Re:Simple solution.... offline backups (Score:2, Interesting)
If your system is compromised for an extended period of time, and you don't know about it, your backups can still get tainted.
i.e. the hacker might modify your backup process in a subtle way, like getting random bits corrupted on the tape, sufficient to make it useless for restoring.
Hardly anyone actually completely tests their backups, to make sure they'll work.
After 6-12 months, the hacker hoses the real DB, not expecting their trap, you try to load from backup from physical media, only to make the situation even worse.
It could take you quite a long time to find a backup that's not hosed (assuming you keep physical media backups older than 12 months)
Re:Why were your backup servers (Score:5, Interesting)
There are a couple ways to 'automate' your backups being "offline" but still automated; not perfect, but better than what they apparently had.
Server on the Internet, with a private subnet behind said server where your backup server(s) are. Internal interface on server must have absolutely zero services running, and the connection must not be routable.
However, backup servers also need to be not kept "online" when not backing up. You can:
* Keep them powered off and only power them on (automatically) when you're doing backups (which occur on a schedule from the main server).
* Keep them on, but with the network interface down.
* If the backup servers use external storage but are kept on, have those on a separate power unit that powers up only on schedule as well.
Personally, this sounds like an "inside job": it's malicious and performed with a great deal of knowledge of the operation. That speaks of a disgruntled employee to me, not some kiddie at it for kicks. There are a lot of steps which can be taken, but realistically, you're not going to be able to deal with something like this unless you've got your data backups completely 'unattached'.
Why were all the backups online? (Score:5, Interesting)
I understand why the most recent backup would be online, but I assumed it was standard procedure to have the remaining archive offline.
Re:It's obvious (Score:3, Interesting)
The one thing about encryption -- key management is just as important as the algorithm used. A business has to figure out how they are going to manage keys. Are they going to use a passphrase that only the backup admins will know, or are they going to use some type of RSA key functionality? If private keys are used, how are they backed up?
There are a lot of factors involved, and one important thing is not depending on one single person as one doesn't depend on one single server. There should be some type of mechanism put in place to deal with an employee gone missing (or even worse, rogue.) For a smaller business, the risks may be low, but its something that should be thought about when planning a backup infrastructure and implementing it.
Re:Why were your backup servers (Score:2, Interesting)
"Their" is not singular, you cretin. Use "his": it's a perfectly fucking good word.
Oh, come off it. [bartleby.com] Quit wasting the world's time.
Re:See also: The classic answer to computer proble (Score:5, Interesting)
First off, lets focus on one thing, if your company is already at a size, complexity, or business need of having backup data electronically replicated offsite (whether this is a true hotsite you're replicating to, or just a method for not rotating tapes through your front door every day), then we're not talking about USB drive data protection schemes here...
OK now, the offsite system should NOT be an administrator access node of your existing backup solution. It should receive replicated data sets and have the ability to operate as a DR server to recover them, but it should be controlled remotely from YOUR site, and should not be able to initiate backups, restores, or make operations changes to another site's DR servers. This is handled by wither firewall rules or user structures.
Second, Physical access at the DR site needs to be as or more strict than at your primary site. If you don;t own the building (or host services through someone like SunGuard) they should have your units in locked racks. Insist that anyone with access has passed Government C2 or higher clearance, and DO NOT give ANYONE outside your organization passwords to those systems.
Next, If the system allows, the user accounts at the hot site should not be the same at any of your other sites.
Next, You STILL NEED TO ARCHIVE DATA, preferably monthly at worst, weekly may be needed depending on your data set size. Replicated systems can corrupt data, and replicated date sets typically only contain your current data set (no rollback copies). Generally, if your replication system is handling moving data automatically, you can leave your local DR copies in your building in a secure area. Daily backups should be archived to local disks so if you have a DR server failue you still have something to recover from (recovery over the WAN is usually not an option). If you have a fire or disasterous event, your offsite systems is your #1 backup, last month/week's archive is your #2.
I've spent a lot of time working with multiple DR companies, and I've supported medium and enterprise class businesses alike (my current employer has 12 mainframes, a couple hundred AIX servers, and over 2000 x86/64 boxes. DR is a BIG thing around here!) Once you enter enterprise class, we're not typically talking "backups" as you know them, more SAN replication and disk write journaling. Recovering 400TB from backups simply isn't an option in a 24-48 hour recovery model! However, if you've got less than 50TB, you can VERY REASONABLY build a great solution out of DPUs (Data Protection Units) made by Unitrends (unitrends.com). They're totally tapeless systems, can share jobs between multiple units at multiple sites, have secure packet level replication technology, and can use 256bit real time encryption as 8GB/minute (per chassis!) with up to 7 concurrent jobs running (per chassis!). a 50TB environment would use 4-8 DPUs (usually departmentalized) conencted to a SAN, and another 1-2 units offsite to handle replication and DR recovery. They also support P2V and V2P recovery and dissimilar recovery. Fully integrated BareMetal backup (they actually invented and patented that one!), and support for 40 operating systems in one multi-site seamless DR solution. You should REALLY look into them. For small sites (less than 1TB data) you can get into one of these units for around $15K with all the bells and whistles (another 10K for replication). There are NO PER CLIENT OR PER AGENT LICENSING FEES, it's ALL inclusive, the only thing you license is storage. Oh, their Data Vault unit can handle up to 40 sites, and each site can be run by a different company with different accounts, and their data can be logically seperated on the vault making it impossible for one customer to interfere with another's data, you can even lock the vault operator out of the solution. This has been validated by the DOD as a secure solution...
Re:Easy fix (Score:2, Interesting)
I'm sorry, but however recommends using an HD as a backup solution, shouldn't be talking about backup.
HDs are NOT backup media.
I just finished up taking about 40 hard drives that I have used for backup over the years and consolidating them onto 2TB drives. Every drive (the oldest is about 10 years old) spun up and all data transferred without a problem.
I've been using hard drives to back up data for years and never had a problem.