Why Do Companies Backup So Infrequently? 403
Orome1 writes "Businesses are on average backing up to tape once a month, with one alarming statistic showing 10 percent were only backing up to tape once per year, according to a survey by Vanson Bourne. Although cloud backup solutions are becoming more common, still the majority of companies will do their backups in-house. Sometimes they will have dedicated IT staff to run them, but usually it's done in-house because they have always done it like that, and they have confidence in their own security and safekeeping of data."
My own backups (Score:2, Interesting)
I run backups at least once per week, and if I make major changes to my system (I self web host), I backup several times per week. This week I lost a 500GB drive. I used ddrescue to recover all but 19MB, although after the filesystem was restored, I lost more than 19MB. Backups saved me huge amounts of time and trouble. The system was down for nearly three days. Data recovery took time, getting a new drive and installing an OS on it, and then rebuilding the system. Its been back for two days. I don't know how companies backup only once per year. Its like they are asking for disaster.
Computers are too reliable (Score:5, Interesting)
The result is that people just expect them to work flawlessly -- not to fail. They also ignore other risks. I put in a machine at some customers a couple of years ago. They did not want to pay extra for backups -- ''yes, we will do that later''. They knew that I configured mirrored (RAID 1) disks. I set up a backup from one part of the disk to another and reminded them every couple of months that they would loose everything if it was destroyed or stolen.
Then a few weeks ago another unit on the industrial was torched -- arson. I have finally pursuaded them ... I am putting in another machine on the far side of their factory that will take a daily rsync, and USB plugin disks that they will backup to weekly & take off site.
These guys are not stupid in what they do professionally, they have an annual turnover of more that £1 million. Why does it take a fire at a neighbour to make them see sense ?
The Benefit of Backups is Non-Obvious (Score:5, Interesting)
Seriously. Most people aren't willing to put in the effort to determine just how bad things would be if they had to resort to their (often inadequate) backups, and therefore they aren't willing to pay the time and capital to get adequate backups.
If you want your company to get better backups, run a simulation of what would happen if something failed. What's the best recovery you could do? What business would you lose? Then calculate the probability of that failure occurring, and be generous.
Re:No wonder they are switching to clouds (Score:3, Interesting)
Surely it completely depends on the size of the company and the importance of the data?
Losing a week's work is *NOTHING* for a small business holder if they have all the normal records in order (it might only take an hour to bring them back up to date), but having a daily backup might actually be a huge chore for them if it involves swapping disks, wearing out tapes, has to be manually initiated, the servers are only on during the working day, etc. Losing a day's work for a 2000-employee company may be a bit more important but they have the resources to backup constantly if necessary.
Personally, if you don't know how important your data is, you have no idea just how often it's worth backing up or whether it's worth backing up at all or, indeed, what needs backing up. Many's the time I've been called into a workplace to restore their backup and they've discovered that the strategy that was fine two years ago was inadequate today because they weren't looking closely enough at what they needed to backup. And if you're not careful, things like "System State" backups can be almost entirely useless to you. Hell, even things like not backing up original install disks can lead to all sorts of problems (i.e. we had weird, obsolete program X that we can't purchase any more installed on the old server, but we can't find the install program to put it on the new server and can't transfer it across because of activation, lack of registry backups or other problems).
A blind "Let's just backup the whole server" isn't an effective backup strategy - it's just ignorance of the task at hand and one that's likely to lead to problems later (what if you upgrade to a larger disk, what if you bring in a new server, what if the server is slower than others and people start moving their data to something that gives them better performance, etc. etc.). Skimming through the thought process of backups by just blanket-backing-up is likely to lead to problems later on, but having an effective way of, say, notifying IT of new things that need to be backed up that people are familiar with and use regularly is worth its weight in gold.
Backup is something to give *extreme* thought to. You want every detail, down to being able to get back to where you are today from TOTALLY DIFFERENT bare metal, no matter what. If you put the thought in, then you are literally working out the costs of a daily vs weekly backup in terms of time, effort, media, storage quotas, etc. as a side-effect anyway and it won't ALWAYS work out in favour of daily backups - especially if you have multiple ways of recreating that data (if you're relying SOLELY on computers for that data, you might want to look at whether that's sensible, for instance, especially if you have 4+ years of record-keeping required by law).
A small company can recreate a week's work in a few days on top of normal work if necessary, and for the hundred-to-one chance of it happening that might well be cost-effective compared to having to up their quotas, keep many more backups (7 times more, or have to cut their possible-history by 1/7th for the same price), change tapes, tie up the servers, etc. A weekend backup would not be unusual at all for a small business.
It also depends very much on the cost of recreating your data. As an IT guy, you can blindly say "We just have to put all our money into backups" but from a business point of view, that's not necessarily cost-effective even if you assume worst-case-scenarios. I'd much rather put more money into regularly working out WHAT needs to be backed up and doing so once a week than being forever complacent and backing up every day. Worst case, you lose a week's work. Same as if there was a flood, or a riot, or a problem with the building, or (in some industries / countries) a holiday, etc.
I specialise in working for schools - bringing back the ones that have slipped into trouble so they are working again - and I see some god-awful messes that I have to clear up. Daily backups don't always save you, especi
Re:To Tape... (Score:5, Interesting)
Slashdot (Score:5, Interesting)
I've always wondered if /. bothers to back up the stories and posts...
Anyone here know?
How about for other 'trivial' sites, like reddit, youtube, etc?
Re:To Tape... (Score:4, Interesting)
But you can combine:
Make local backup to disc, and use cloud-based backup as the off-site-storage.
This was you easily get near-zero-administration. You get quick restores for the more common scenarios (server died, user accidentally deleted something important, disc crashed).
Yes you get slow restores for the uncommon cases (those where both the primary storage, and the on-site-backup are fubared) this happens if, for example, a fire destroys the building or thieves steal everything that looks electronic and expensive - but in these scenarios it's going to take some time to get back on the air anyway, aslong as you ain't got atleast two physically disjoint datacenters.
If you ain't got huge amounts of data, this is cheap. If you *do* have huge amounts of data, it can get expensive and inacceptibly slow, in that case you might need a different solution. We've got around a terabyte in the cloud (it automatically de-duplicates, so the same file is only ever stored once), and this costs us $750/year which is cheap enough to be a no-brainer. (it'd be different if we needed to backup hundreds of terabytes though)
We've got a 100Mbit link, so restores happen reasonably fast, but a complete restore from nothing of everything, would still take time.
Re:To Tape... (Score:5, Interesting)
Disclosure: I work in backup and recovery for a living. I've been doing TSM [wikipedia.org] professionally for the past six years.
So let's look at this idea that portable HD is cheaper and faster. I've managed backup systems that were backing up tens of terabytes every day. Let's say three 3 TB HDDs for the sake of discussion. Let's say five weeks' worth of data retention. That's 5*7*3 = 105 hard drives that need to be managed - call it 110, to take into account the fact that there are going to be a few going off and onsite at any given point in time.
How reliable are those hard drives? How long until they fail? Bear in mind when considering your answer that hard drives are designed to be stuck inside a chassis that stays put (for the most part) - pulling them out of the system and lugging them halfway across town is not within their design goals. And then you need to worry about the cost of a hot-swap chassis so you can pull the drives out, and shove them back in, every day. Oh, and don't forget that most plugs have a limited life span - probably in the hundreds of swaps, maybe thousands (I don't know, and I'm happy to hear solid data about this.)
And how long does it take to fill up a hard drive, anyway? Take the WD Green 3 TB as an example: 110 MB/s (source [techreport.com]), equals about 7.5 hours, best case. Sure, you can throw more hard drives at the problem in parallel, but that just exacerbates the whole question about reliability in transit. The Hitachi 3 TB is faster - 207 MB/s [techreport.com] - which takes about four hours to fill up. That's best case scenario, based upon the maximum data transfer rate - guaranteed it's going to slow down as the drive fills.
Now consider tape. Consider wikipedia's information [wikipedia.org] on LTO when reading this. LTO4: 120 MB/s native, up to 300 MB/s compressed (two hours to fill). LTO5: 140 MB/s native, 350 MB/s compressed (three hours to fill). Pretty damn reliable in transit; they're designed to sit happily in their little plastic shells, in a box, and get thrown around (not quite, but they can certainly take more punishment than a hard drive can.) Capacity per cartridge: 800 GB/1.5 TB native (4/5); 1.6 TB/3 TB compressed. If you have money to burn, you could go for Oracle's T10000C drive: 5 TB native capacity, 240 MB/s native throughput (and that's before you get into the whole question of compression.)
Now let's get onto the whole subject of financial data, Sarbanes-Oxley, and WORM media (so you know the data hasn't been altered since it was written out) ...
Sure, tape sucks. It has major issues when data is scattered all over the place; mount time takes a while; and the drives need regular love and attention. But here's the thing - it survives today because it's better than the alternatives in its niches, and trust me, there's plenty of niches where tape fits in far better than the alternatives.
If you only have a couple of TB of data to backup, the cost of setting up tape infrastructure will probably not be worth it. But when you're talking hundreds of TB - or better, petabytes of data (don't laugh, one client I did work for had over 2 PB of data in their tape library) - the cost equation swings over pretty damn fast. Tape is not dead. Far from it. I can't see the likes of IBM investing in developing LTO6 and LTO7 if there was no use for it. And why would Oracle sell a tape library that scales to 100,000 slots [oracle.com] if there's no demand for it? It's not about how to get the most bytes for your dollar - it's also about reliability, and that gets down to the usage. If I suggested portable hard drives to the clients I do work for, I'd be out of a job - because they simply won't cut it for their needs.
Re:To Tape... (Score:5, Interesting)
I've had similar experience, but I've found that it swings back toward disk in a big way on the low-end. If you can fit your data onto a single 2TB disk, then it's much more reliable than tape, and cheaper too.
Tape is also a lot less reliable outside data centres, because practically nobody designs those drives to survive the dust in a typical office environment. I've seen about 50% or more of the tape systems out in the field fail at least once a year, but I've never seen a disk based backup system fail. Note once, not ever.
Also, tapes aren't as robust as people think they are. One of the hardest IT troubleshooting jobs I've ever worked on was a tape system that had regular failures. Turned out that the IT guy liked to throw the tapes up in the air (only about a foot or two), and then catch them. He didn't drop them or anything, but that was enough to cause regular backup failures. That's less robust than a (powered off) disk drive.
I've got a feeling that tape reliability numbers are massively exaggerated in marketing materials. For example, I once had a tape getting repeatedly loaded, read a little bit, and then unloaded overnight because of a software bug. It was destroyed. Think grooves etched into the plastic casing, and the tape worn to the point of transparency. That got me thinking, and I looked up the numbers.
People get confused by numbers like "1,000,000 passes" in the specifications. You have to read the full version: "1,000,000 passes on any area of tape, equates to over 20,000 end to end passes/260 full tape backups". People forget that LTO makes many passes per backup, so suddenly you're down to a three digit number of backups instead of the huge sounding million they start with in the brochure. Throw in verify-after-backup, and it's only 130. If you back up daily, that's just over four months before your tapes are worn out, according to the spec. Meanwhile, even consumer grade hard drives can last for years, even decades.
Re:To Tape... (Score:5, Interesting)
The problem is you think of backups as something that has to be ran as a batch.
Our backup is different. Our SAN uses the cloud natively. It does deduped snapshots and clones to the cloud. All of our data is in the cloud, we have about 8TB of data total, but our cloud costs thanks to dedup are around 700 a month.
We then do traditional backups to local disk for fast, short term recovery. Our SAN is built in such a way that if it was to die, a new one could be dropped in and immediately start using the cloud while it slowly caches everything back down to disk. So in essence, our SAN is our backup.
As a company with almost everything virtualized, we don't need bare metal restore, we just need those vmdk files and to be available. Backups are a lot easier than when I was here 5 years ago, and our recovery rates during testing are a lot higher.