How To Prevent Being Hacked Via Backups?

Popsikle writes "A few days ago one of the Web's largest hosting discussion forums was supposedly hacked via their backup servers. From the story: 'We've since learned that this very deliberate, sophisticated and calculated hack against Web Hosting Talk was carried out by gaining access to our offsite backup servers. From our backup servers, the hacker gained access to the WHT db server. The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.' What sort of security do you put on your backup infrastructure? Looking at your backup solution could you be completely taken down by either someone obtaining a backup or accessing your backup servers? What sort of recommendations does everyone have for this not to happen?"

Obvious Oversight by WHT

[Revotron]

There was a very blatant oversight and an unfortunate assumption on the part of WHT and iNET Interactive.

They quite obviously overlooked the fact that the WHT servers (and ONLY the WHT servers) would ever need routine access to the backup servers. Therefore it was an obvious security hole that could have been plugged by restricting traffic through iptables to only iNET-affiliated IPs. Any teleworkers who needed access should simply use a VPN to iNet's offices if they really need access to the backup systems. If under some extreme circumstance (such as the loss of a database) an outside party needs access to the backup servers, the system admin can then add an exception under iptables.

And on that note, the other incredibly thoughtless assumption was that any traffic coming from the backup servers would be approved traffic. So once the attacker gained access to the backup servers, the database servers were one insecure hop away.

I think this proves the following very important points to the entire IT industry:
1) Internal infrastructure should remain just that - internal! Restrictions should always be put in place as to who can (or can't) access a system.
2) No traffic can be guaranteed authorized or authentic. It's one thing to add an SSH keyfile to your home servers, but in an enterprise environment everything must be highly scrutinized. It's no longer a matter of protecting systems from users - it's now a matter of protecting systems from other systems as well.

I was personally affected by the loss of information at WHT and while it's annoying, it's a fact of life and can't be undone. All that's left now is to pick up the pieces, secure the site as best they can and move on with lessons learned.

Re:Tachikoma

[KahabutDieDrake]

Once they gain individuality, they tend to kill themselves in spectacular ways. I'd prefer a security system that didn't self destruct on top of the intruder, at least assuming there was another option.

Re:Easy fix

[magisterx]

I would respectfully beg to differ. For my personal computer, I have two layers of backup, one goes to an external harddrive and works quite well to protect me from harddrive failure or being stupid and overwriting the wrong file. The other is burning a set of DVDs every now and then and store elsewhere, which protects me from physical destruction of my house.

At my last job we had 3 layers of backups; the first was to a large harddrive (actually a RAID away, but still an array of harddrives.) Then from there we made tape backups for short term storage and less frequent tape backups for long term offsite archival. But again, the harddrives where the first layer.

In short, I find harddrives make an excellent first layer of backup protection. They are faster and (depending on setup) easier to use than most other backup solutions, and very affordable.

Of course, I would never recommend relying on harddrives as your only layer of backups when dealing with mission critical data, but then I would never rely on any one layer of backups for truly mission critical data. Harddrives probably provide one of the best first layers in that case.

If you are dealing with less critical data and working on a budget then you may be able to accept just one layer of backups, and in that case then which way you choose to go depends on a number of factors but harddrives are a strong contender even there.

Re:Easy fix

[morcego]

Ok, there were several replies to my post and, since I'm not going to reply to all of them, I decided to pick one that is clear and definitively worth discussing with.

I do agree with your backup strategy. It is sound. I use that one some sites, having backup made to tapes, and a secondary storage area (or a pre-backup staging area) on a NAS, for fast recovery of trivial files. Restoring files from the NAS is usually fasted, as you stated, than from a tape.

Using BOTH HDDs and tapes to supplement each others, specially for added redundancy is very sound.

However, HDDs don't replace tapes. They are too prone to fail, specially if left powered off for long periods of time. It is a catch 22 if I ever saw one. If you keep them powered on, they will fail sooner. If you keep them powered off, they will also fail sooner (than tapes). Also HDs are much more sensitive than tapes to storage condition. Although if you are worth your salt, you probably have a storage vault/room/facility with controlled humidity, temperature, EM-free and all that.

Not can stop you from calling your HDD "a backup", the same it we can't stop you from calling it "Bob".

Yes, as a side note, (newer) HDDs are more reliable than some old tape backup technologies, like DDS. At least on my experience. But if you are running backups on DDS tapes, you are in enough trouble already.

For those that want to argue that tape technologies change too much, I would like to invite them to read the data from a RLL HDD I have here on my cabinet. I also have a MFM HDD, but I don't want to push the issue too much.

Re:Easy fix

[Cramer]

HDs are not archive media. They don't fair too well left sitting on a shelf (unpowered) gathering dust for a few years. Magnetic tapes can (and DO) last decades in storage. Yes, they are slow and far more expensive, but they are, without a doubt, the best option for offline archives and backups.

In fact, most "cost effective" (read: cheap) IDE/SATA drives tend to last about 3 years in normal/typical operation. They become highly unreadable if left powered off for 6+ months. Tapes, on the other hand, last for decades.

Re:Tapes Are Rubbish

[UnrefinedLayman]

Tapes are rubbish. Tape is expensive and unreliable. Anyone that tells you otherwise is selling the stuff.

I hope nobody confuses your inability to think beyond your world with fact, because this is the hugest crock of shit.

I have about 12 TB of data that I am required to be able to restore to any day within the last five weeks. One week of backups = 20 TB. In addition, I must:

1) always have at least four weeks' worth of data in different building on the same site
2) always have at most one week's worth of data in the rack
3) always have at least one full backup from within the last five weeks at an off-site storage facility
4) retain every fifth backup for one year
5) retain every 52nd backup for seven years

What are my options?

Option 1 (your option): Buy a minimum of 442 TB of additional disks (102 TB for five weeks of storage [1 & 2 above]; 200 TB for [4]; 140 TB for [5]; 37 times more storage than we currently use), plus the SAN, plus the network, power, and cooling systems to support it. When rotating media, physically remove the drives and carry them around. Carefully. Because one backup weighs thirty pounds. Media cost: $132,000 ($300 per); SAN cost: $50,000.

Option 2 (my option): Buy a tape library and 442 TB of tapes (one library, 368 tapes). When rotating media off-site, throw the tapes into a bag. Carelessly. Because they weigh 15 pounds and don't break. Media cost: $20,600 ($56 per); Library cost: $20,000

Looking at the last 600+ rotations I've done for various systems in the last five years, the failure rate is far, far, far lower than hard drives already; when you take into account the fact that 20% of all tapes are transported 60 miles per year and 50% of all tapes are transported between local sites at least twice per month, the failure rate of hard drives would be through the roof. Further, while I have had tapes fail during a write operation, I have never had a tape fail during a data restore, even on tapes overwritten hundreds of times and tapes stored for years. Perhaps your backup administrator has never heard of data verification or proper media storage, but if he had perhaps you wouldn't have experienced all those repeated failures of your backup system.

Try being a backup administrator, not just playing pretend like you are now, before being so glib.