Why Do Companies Backup So Infrequently? 403
Orome1 writes "Businesses are on average backing up to tape once a month, with one alarming statistic showing 10 percent were only backing up to tape once per year, according to a survey by Vanson Bourne. Although cloud backup solutions are becoming more common, still the majority of companies will do their backups in-house. Sometimes they will have dedicated IT staff to run them, but usually it's done in-house because they have always done it like that, and they have confidence in their own security and safekeeping of data."
To Tape... (Score:5, Insightful)
Portable HD is cheaper and faster, even for stacks of them. Small businesses may be using a bunch of these in place of tapes.
Easy (Score:5, Insightful)
risk (Score:5, Insightful)
Most companies have no risk management, and no clear picture of the risks their business faces.
The result of "intuitive" risk-non-management is that the usual human flaws have full impact. Basically, aside from a narrow middle ground, all risks are wrongly estimated.
backup often, and respect the 'rm' (Score:5, Insightful)
especially when combined with 'find' and 'xargs', in what is supposed to be a simple task.
If you don't, you'll do something like what i just did ("worst typo in a decade"): you see, i was trying to update emacs and wanted to purge all the .elc files from ~/.emacs.d
Unfortunately, through a bad typo, some miss-applied keyboard shortcuts, and rushing through without mounting a scratch monkey... what actually ran was effectively "find ~/.emacs.d | xargs rm".
accidently deleted the 'grep'. Oops. 15+ years of elisp/etc destroyed.
Was it backed up? Nope! Been meaning to check it all into git, but always put it off as a "minor, unimportant" task I'd get to later. Of course, we all think that way up until the disaster hits...
*sigh*
Re:No wonder they are switching to clouds (Score:2, Insightful)
Re:To Tape... (Score:4, Insightful)
Multiple backups to multiple media.
More than once I've seen tape recovery atempts fail because a streamer didn't work correctly, the tapes had manufacturing faults or their software of choice was nothing but a useless piece of shit.
Re:Easy (Score:4, Insightful)
Document the hell out of that!
See if you can cause a server to be down. When the CEO screams tell him such and such wanted to save $400 and didn't think it was worth it save $40,000 an hour on lost productivity to save $400 now. Sorry that had to happen during that $ 1,000,000 deal you lost. Heads would roll faster thanyou can say fired. ... unfortunately, this makes a cloud all the more attractive that could end your own job :-(
Companies are really stupid today! Normally I do not use exclamation marks, but the latest scaffle in Thailand where every single hard drive is made on earth in an all your eggs in one basket to save on 5% economies of scales cost, to companies producing junk, and now this drives me crazy. It bothers me because so many of us are out of work and these companies are penny wise but certainly dollar dumb. To me no tape backup in any fortune 2,000 company is the equilivent of having no fire insurance. A single disaster could cost you everything and it is so cheap when you rake in millions.
Re:Who uses tape any more? (Score:5, Insightful)
RAID is not a replacement for a backup.
RAID will safeguard you against the failure of a single disk (if and only if you monitor the system and replace disks as they fail), but backup will give you back your data as it was before your application destroyed it or your user deleted it. That is something completely different.
Re:To Tape... (Score:5, Insightful)
Complacency (Score:5, Insightful)
To most PHBs a computer is a toaster. That's right, an appliance. Nothing more. Most have no idea of the nursemaiding a computer needs, or how vulnerable company data is when it's reduced to 0s and 1s. Until the "unthinkable" happens and the toaster breaks.
Now. I used to be in the biz of disaster recovery. It was lucrative when consumers who valued their potentially lost data found me and asked me "Can you do it?" and I said "Let's have a look". My success rate was something like 40%. Surprisingly (for me anyway), my better corner was recovering data from physically damaged flash drives. Out of several dozen of those I only had one that I couldn't recover anything from. Hard disks are a different animal, and the whole thing can be frustrating when you sit back for a minute or five and consider how much easier your work would be (and how much hardware you can sell) if your client had the benefit of hindsight and someone around who knew the shit of which he spoke - so a week-long recovery project that may or more than likely may not work turns into a two hour exercise in restoring on new hardware and sending the happy client on his way. Now, that is an exercise in getting repeat business through recommendation.
As far as "accidental" deletion of data: there is only so much you can do to protect the stupid from themselves - on a network share, for instance, you can deny users the right to delete files. Job done. In an environment that is supposed to be secure, that's a good start. VSC is another handy tool but you don't need to tell the stupid that their fuckups are (sort of) covered - it breeds complacency and does little to nothing to train responsibility.
Re:To Tape... (Score:5, Insightful)
well, doh. do it to tape once a year. it's unlikely all hd's would fail at the same time.
checking _what_ was backed up on the other hand.. how many companies check that..
Re:Offsite backups (Score:5, Insightful)
Offsite backups are what a lot of companies don't do. They might back up to tape, but the tapes are stored in a pile next to the server. And they never test them.
Agreed. One of the many phrases in my litany on customers backing up is "Yes, that safe is fireproof. For paper. Plastic melts at a much lower temperature than paper burns." Not only do not enough companies run backups on a regular and timely basis, but too many of the ones that do run backups don't see the need for having the storage media offsite.
Re:To Tape... (Score:5, Insightful)
The thing is though, tape *sucks*.
The most common reason for needing a restore, is accidental deletion. With modern backup-systems using online disc, the user can in this case simple open his backup-client, find the file, and click "restore" - time elapsed 2 minutes, help needed, none.
Tape means talking to IT, and wait for hours, at significant personnel-cost. Unless there's a fancy automatic tape-switching-robot kind of deal, but if there is, the price is no longer $30/TB.
Yes you need off-site-backup in addition, and it's acceptable for that one, to have higher latency, so perhaps tape is okay for that.
Re:No wonder they are switching to clouds (Score:5, Insightful)
I hope you're a troll, because if you're not, this is just about the most insanely fucked up attitude to data protection I've ever seen.
A blind "Let's just backup the whole server" isn't an effective backup strategy ...
If I was your boss I'd fire you on the spot for being that monumentally stupid. The cost ratio of an additional ~20GB of tape compared the enormous cost to the business if the server can't be restored quickly and reliably is staggering. Think $1 saved while risking millions of dollars of potential lost work! (1.6TB LTO5 tape = $80)
a) "what if you upgrade to a larger disk" -- there is no backup system on Earth that can't restore to bigger disks. Most systems can restore to smaller disks too.
b) "what if the server is slower than others and people start moving their data to something" -- you back up both of them. Restoring too much is almost never as bad as restoring too little.
c) "just blanket-backing-up is likely to lead to problems later on" -- no, it doesn't. Achieving 100% coverage ensures that no matter where you data was on a server (or which server), it's on a tape somewhere.
d) "notifying IT of new things that need to be backed up" -- have you met humans? This never happens reliably, and can't ever be made reliable.
Back in the real world, a 100% complete backup of a typical Windows server can be restored without knowing the password, to dissimilar hardware (even virtual machines), and without needing the "original install disks". When it's done, it'll boot up, maybe reboot once or twice to fix up its drivers, and then your server is back, working as it did before. Compare that to a "data only" or partial backup. Now suddenly you're chasing down design documentation, passwords, IP addresses, software, serial numbers, and you haven't even started to restore anything yet. The clock is ticking, and the customer is breathing down your neck.
A week's work should take no more than two weeks at ABSOLUTE maximum to recreate
Recreate from what? Memory? Including data that was 100% electronic, and never seen by a human? How do you recreate your emails? How do you recreate your audit logs? How do you type back in non-textual data like digital images or audio recordings? How would you even know what's missing?
I'd rather have a decent monthly, than an imperfect daily
That's a false dichotomy. The total data stored is the same, you're just altering the frequency. The same amount of storage is needed, the same bandwidth is needed, and it ends up costing the same.
I'd rather have three backups a day than monthly backups. Losing a day of work could mean a contract fails to go through. I've been in a position twice now where users have come to me literally crying and begging to retrieve a document they only started working on that morning that they had deleted accidentally... at 8pm, minutes before a deadline for a multi million dollar deal. After experiences like those I've often set up incremental backup frequencies as rapid as 15 minutes.
So lets recap... your sum total DR experience is you once walked into a poorly supported environment, and gave them some even worse advice, without ever being in a position to be responsible for an actual real world recovery.
Well, take some advice from someone who's restored terabytes of data, and was responsible for the protection of over a petabyte spread across thousands of servers at over a dozen organisations:
#1 There are no time machines -- you cannot go back in time to fix a mistake in your backup strategy after a disaster. It's too late. You've fucked up, it's your fault, and you can never, ever, fix it.
#2 Back up everything -- I love genius IT folk who like to shave 1% off their backup times by excluding those useless temp and log files, also excluding 'useless junk' like their database transaction logs in the process.
Re:My own backups (Score:4, Insightful)
The purpose of RAID is to avoid most of the cases where you would have had to restore from a backup, such as a drive failing.
Re:To Tape... (Score:5, Insightful)
The thing is though, tape *sucks*.
The most common reason for needing a restore, is accidental deletion. With modern backup-systems using online disc, the user can in this case simple open his backup-client, find the file, and click "restore" - time elapsed 2 minutes, help needed, none.
Tape means talking to IT, and wait for hours, at significant personnel-cost. Unless there's a fancy automatic tape-switching-robot kind of deal, but if there is, the price is no longer $30/TB.
Yes you need off-site-backup in addition, and it's acceptable for that one, to have higher latency, so perhaps tape is okay for that.
What you describe sounds like a tape-per-server solution, but anyone using virtualization or even having a reasonable size will use an enterprise backup solution that's user friendly regardless if the back end is disk, tape, or both. We use TSM to tape libraries of Ultrium5 drives (and disk pools for a backup landing place/1st day restore). Need to restore files or directories? Open up the local web client, click what you want in the GUI (and potentially a point in time you want to restore it from), and it's automagically done in 1-2 minutes. Regardless if ti's last night's backup or something from 3 years ago. Most of that time is the robot loading the tape(s) and seek time - getting data off the tapes is fast.
We back up about 5TB of changes daily, and reclaim about 3.5TB of old no-longer needed data daily (much needs to be kept for extended periods for compliance issues), with over 2PB in-use storage. Power/cooling costs for tape barely move the needle, that much disk would have a large impact on our datacenters even using deduplication.
Also, we have both on-site (for speed of restore) and off-site copies of everything. Easy to do, the tapes backed up at night make copies automatically during the day. This isn't a replacement for disk replication to alternate sites for site disaster, it's for normal restores and keeping everything we need for compliance reasons safe.
I'm not saying disk is bad - disk is good. But tape is good too, and it's a different tool - fits a different niche than disk. Hammer and screwdriver will let you tackle more jobs then either one by itself.
Re:To Tape... (Score:4, Insightful)
FWIW: We use the cheapest 2.5" laptop drives available, in the cheapest bus-powered 2.5" USB enclosures we can get our hands on (I think the last round of them cost us $5, each).
Did you look at using trayless hot-swappable drive racks? They're very handy, you just plug the bare drive in and away you go at full SATA speed.