Why Do Companies Backup So Infrequently? 403
Orome1 writes "Businesses are on average backing up to tape once a month, with one alarming statistic showing 10 percent were only backing up to tape once per year, according to a survey by Vanson Bourne. Although cloud backup solutions are becoming more common, still the majority of companies will do their backups in-house. Sometimes they will have dedicated IT staff to run them, but usually it's done in-house because they have always done it like that, and they have confidence in their own security and safekeeping of data."
To Tape... (Score:5, Insightful)
Portable HD is cheaper and faster, even for stacks of them. Small businesses may be using a bunch of these in place of tapes.
Re:To Tape... (Score:5, Informative)
Indeed. My own [small place of business] does its backups this way with Acronis:
-
Every day, a drive is plugged in.
Every evening, an incremental backup is performed on that drive that is consistent with whatever the backup drive already had on it. (It just takes a few minutes for Acronis to get this done.)
After that it is removed and taken off-site. A different drive is plugged in the following day.
Fridays are special, in that the drive gets automagically erased, and a full backup is performed mid-day. But then the evening incremental ritual is the same as any other day of the week.
-
Losing a backup disk is, at most, 24 hours worth of loss, and only then if it is coincident with losing the main system.
FWIW: We use the cheapest 2.5" laptop drives available, in the cheapest bus-powered 2.5" USB enclosures we can get our hands on (I think the last round of them cost us $5, each). And we test our backups randomly, whenever the accountant decides he wants to see what things looked like last week/last year. Acronis then cheerfully does a bare-metal restore from [random backup drive] onto our spare server-box, with 100% success.
Now: We're not a media company. We don't have tens of TB of changing data to back up on a daily basis. But most other small(ish) companies don't either...
Who needs tapes?
Re: (Score:3)
That doesn't make sense, if you are using a new drive each day, how does the friday wipe come into play? Where do you store your thousands of drives? And how the hell do you manage to pick up drives for $5?
If you are not using a new drive each day, how do you recall a backup from a year past when a drive is wiped every friday?
Re:To Tape... (Score:5, Informative)
There are many different backup strategies depending on what you are backing up and if your backups are designed for disaster recovery or archival of data.
Making some presumptions here to answer your questions:
They use a rotation of 5 or (7 drives if including weekends). Monday's drive is always the same, as is Tuesdays, Wednesday, etc. Each drive is kept offsite and only that day's drive is brought onsite while others remain offsite, or are taken back offsite.
If on Thursday the server goes down, they just request Wednesday's drive for the restore. If that backup was unsuccessful or the drive just happened to fail, then they lose Wednesday's business and go back to Tuesdays. Repeat until they get a successful restore.
Fridays basically are not an incremental backup. Well, the evening is, but the midday backup is a full backup and then is incrementally updated from the 1/2 day of business that night.
With this scheme, they can't restore a backup from a year ago. Depending on what is being backed up, they may not have to go back a year. Accounting information for instance from a year ago may be in the current backup set. If you are looking to restore something that was corrupted or deleted at an unknown point and wouldn't be in the current backup, then it would just take 1 disk out of the rotation once a month for instance and replace it with a new drive. A previous company I worked for recommend this strategy with point of sale systems we sold. We gave them a box of 10 backup tapes. 6 nightly backup tapes and 4 weekly tapes. The weekly tapes were in a 4-week rotation while the nightly were in a 1 week rotation. If they wanted to do a monthly rotation they could purchase additional tapes.
The $5 wasn't for the drive, it was for the enclosure and/or drive adapter.
Re:To Tape... (Score:4, Insightful)
FWIW: We use the cheapest 2.5" laptop drives available, in the cheapest bus-powered 2.5" USB enclosures we can get our hands on (I think the last round of them cost us $5, each).
Did you look at using trayless hot-swappable drive racks? They're very handy, you just plug the bare drive in and away you go at full SATA speed.
Re: (Score:3)
The bare drive to read/write that $25 800GB tape will have cost more by itself than our entire backup system (including drives, enclosures, and software), and presents itself as a single point of failure unless we buy more than one of them.
How, then, do you suppose that tape is cheaper or more reliable than a pile of inexpensive redundant hard drives?
I recognize that your needs may be different. Please recognize that mine may be as well.
Re:To Tape... (Score:5, Insightful)
Re:To Tape... (Score:5, Informative)
Perhaps for *very* small businesses with not much data, or where the restore time is not a critical factor.
Even with our 32Mbit connection at work it would take about 40 hours in transfer time alone to get our data "out of the cloud" in a disaster scenario. (And part of that bandwidth would still be needed to keep other operations running, so it might even take longer) You can perhaps make small, incremental backups into the cloud, but when you need the data in an emergency you need it as fast as possible. Restore from tape took two hours. (We had a chance to do it "for real" earlier this year ;-P )
Re:To Tape... (Score:4, Interesting)
But you can combine:
Make local backup to disc, and use cloud-based backup as the off-site-storage.
This was you easily get near-zero-administration. You get quick restores for the more common scenarios (server died, user accidentally deleted something important, disc crashed).
Yes you get slow restores for the uncommon cases (those where both the primary storage, and the on-site-backup are fubared) this happens if, for example, a fire destroys the building or thieves steal everything that looks electronic and expensive - but in these scenarios it's going to take some time to get back on the air anyway, aslong as you ain't got atleast two physically disjoint datacenters.
If you ain't got huge amounts of data, this is cheap. If you *do* have huge amounts of data, it can get expensive and inacceptibly slow, in that case you might need a different solution. We've got around a terabyte in the cloud (it automatically de-duplicates, so the same file is only ever stored once), and this costs us $750/year which is cheap enough to be a no-brainer. (it'd be different if we needed to backup hundreds of terabytes though)
We've got a 100Mbit link, so restores happen reasonably fast, but a complete restore from nothing of everything, would still take time.
Re:To Tape... (Score:5, Informative)
Re:To Tape... (Score:5, Interesting)
The problem is you think of backups as something that has to be ran as a batch.
Our backup is different. Our SAN uses the cloud natively. It does deduped snapshots and clones to the cloud. All of our data is in the cloud, we have about 8TB of data total, but our cloud costs thanks to dedup are around 700 a month.
We then do traditional backups to local disk for fast, short term recovery. Our SAN is built in such a way that if it was to die, a new one could be dropped in and immediately start using the cloud while it slowly caches everything back down to disk. So in essence, our SAN is our backup.
As a company with almost everything virtualized, we don't need bare metal restore, we just need those vmdk files and to be available. Backups are a lot easier than when I was here 5 years ago, and our recovery rates during testing are a lot higher.
What about the long term? (Score:3)
Re: (Score:3)
And large businesses have even less internet bandwidth in ratio to the amount of data that needs to be backed up.
Re: (Score:2)
Re: (Score:3)
I am aware that Hard Drive prices have recently increased due to the flooding but even before that, back when HDD prices were low, tapes still under-cut them.
Re:To Tape... (Score:5, Interesting)
Re: (Score:2)
Where? A quick google shows $130-$160 for the tape which is actually 500GB but MIGHT store 1TB if the data is compressible enough. For $130, I can get a 1-2TB hard drive that will store an ACTUAL 1-2TB of data, more if I use compression.
Re: (Score:3)
Re: (Score:3)
Except for the little part where the LTO5 tape drive costs $5000 - $10,000?
Re:To Tape... (Score:5, Insightful)
The thing is though, tape *sucks*.
The most common reason for needing a restore, is accidental deletion. With modern backup-systems using online disc, the user can in this case simple open his backup-client, find the file, and click "restore" - time elapsed 2 minutes, help needed, none.
Tape means talking to IT, and wait for hours, at significant personnel-cost. Unless there's a fancy automatic tape-switching-robot kind of deal, but if there is, the price is no longer $30/TB.
Yes you need off-site-backup in addition, and it's acceptable for that one, to have higher latency, so perhaps tape is okay for that.
Re: (Score:3)
Amusingly some local tech staff members at a certain norwegian gas processing plant do local hd backups of their network shares and such for this very reason... It takes so damn long to get a backup restored that they have another level of backups locally to avoid having to ASK the central IT for help...
Ugh..
Memory hierarchy (Score:3)
Re: (Score:3)
Re:To Tape... (Score:5, Insightful)
The thing is though, tape *sucks*.
The most common reason for needing a restore, is accidental deletion. With modern backup-systems using online disc, the user can in this case simple open his backup-client, find the file, and click "restore" - time elapsed 2 minutes, help needed, none.
Tape means talking to IT, and wait for hours, at significant personnel-cost. Unless there's a fancy automatic tape-switching-robot kind of deal, but if there is, the price is no longer $30/TB.
Yes you need off-site-backup in addition, and it's acceptable for that one, to have higher latency, so perhaps tape is okay for that.
What you describe sounds like a tape-per-server solution, but anyone using virtualization or even having a reasonable size will use an enterprise backup solution that's user friendly regardless if the back end is disk, tape, or both. We use TSM to tape libraries of Ultrium5 drives (and disk pools for a backup landing place/1st day restore). Need to restore files or directories? Open up the local web client, click what you want in the GUI (and potentially a point in time you want to restore it from), and it's automagically done in 1-2 minutes. Regardless if ti's last night's backup or something from 3 years ago. Most of that time is the robot loading the tape(s) and seek time - getting data off the tapes is fast.
We back up about 5TB of changes daily, and reclaim about 3.5TB of old no-longer needed data daily (much needs to be kept for extended periods for compliance issues), with over 2PB in-use storage. Power/cooling costs for tape barely move the needle, that much disk would have a large impact on our datacenters even using deduplication.
Also, we have both on-site (for speed of restore) and off-site copies of everything. Easy to do, the tapes backed up at night make copies automatically during the day. This isn't a replacement for disk replication to alternate sites for site disaster, it's for normal restores and keeping everything we need for compliance reasons safe.
I'm not saying disk is bad - disk is good. But tape is good too, and it's a different tool - fits a different niche than disk. Hammer and screwdriver will let you tackle more jobs then either one by itself.
Re:To Tape... (Score:4, Informative)
The most common reason for needing a restore, is accidental deletion.
It is, but the most common use for tape is compliance, Many companies, and all public companies have compliance issues, SarBox, HIPAA, GLB, etc. Most require data retention of many years. In some medical settings as much as 21 years. Do you want to keep that all on spinning disks? or on CHEAP tape sitting in a box at Iron Mountain? In my environment, and that of many large companies, we no longer measure terrabytes, except at the individual database level. We manage several petabytes.
disk based solutions are operationally ideal. Thats why products like Avamar and Data Domain do so well. But for large, long term, low access, storage, tape is still king.
Re: (Score:2)
If you are using Amazon for backup you are doing it wrong. Amazon has a different purpose is much to expensive for that.
Re:To Tape... (Score:5, Interesting)
Disclosure: I work in backup and recovery for a living. I've been doing TSM [wikipedia.org] professionally for the past six years.
So let's look at this idea that portable HD is cheaper and faster. I've managed backup systems that were backing up tens of terabytes every day. Let's say three 3 TB HDDs for the sake of discussion. Let's say five weeks' worth of data retention. That's 5*7*3 = 105 hard drives that need to be managed - call it 110, to take into account the fact that there are going to be a few going off and onsite at any given point in time.
How reliable are those hard drives? How long until they fail? Bear in mind when considering your answer that hard drives are designed to be stuck inside a chassis that stays put (for the most part) - pulling them out of the system and lugging them halfway across town is not within their design goals. And then you need to worry about the cost of a hot-swap chassis so you can pull the drives out, and shove them back in, every day. Oh, and don't forget that most plugs have a limited life span - probably in the hundreds of swaps, maybe thousands (I don't know, and I'm happy to hear solid data about this.)
And how long does it take to fill up a hard drive, anyway? Take the WD Green 3 TB as an example: 110 MB/s (source [techreport.com]), equals about 7.5 hours, best case. Sure, you can throw more hard drives at the problem in parallel, but that just exacerbates the whole question about reliability in transit. The Hitachi 3 TB is faster - 207 MB/s [techreport.com] - which takes about four hours to fill up. That's best case scenario, based upon the maximum data transfer rate - guaranteed it's going to slow down as the drive fills.
Now consider tape. Consider wikipedia's information [wikipedia.org] on LTO when reading this. LTO4: 120 MB/s native, up to 300 MB/s compressed (two hours to fill). LTO5: 140 MB/s native, 350 MB/s compressed (three hours to fill). Pretty damn reliable in transit; they're designed to sit happily in their little plastic shells, in a box, and get thrown around (not quite, but they can certainly take more punishment than a hard drive can.) Capacity per cartridge: 800 GB/1.5 TB native (4/5); 1.6 TB/3 TB compressed. If you have money to burn, you could go for Oracle's T10000C drive: 5 TB native capacity, 240 MB/s native throughput (and that's before you get into the whole question of compression.)
Now let's get onto the whole subject of financial data, Sarbanes-Oxley, and WORM media (so you know the data hasn't been altered since it was written out) ...
Sure, tape sucks. It has major issues when data is scattered all over the place; mount time takes a while; and the drives need regular love and attention. But here's the thing - it survives today because it's better than the alternatives in its niches, and trust me, there's plenty of niches where tape fits in far better than the alternatives.
If you only have a couple of TB of data to backup, the cost of setting up tape infrastructure will probably not be worth it. But when you're talking hundreds of TB - or better, petabytes of data (don't laugh, one client I did work for had over 2 PB of data in their tape library) - the cost equation swings over pretty damn fast. Tape is not dead. Far from it. I can't see the likes of IBM investing in developing LTO6 and LTO7 if there was no use for it. And why would Oracle sell a tape library that scales to 100,000 slots [oracle.com] if there's no demand for it? It's not about how to get the most bytes for your dollar - it's also about reliability, and that gets down to the usage. If I suggested portable hard drives to the clients I do work for, I'd be out of a job - because they simply won't cut it for their needs.
Re:To Tape... (Score:5, Interesting)
I've had similar experience, but I've found that it swings back toward disk in a big way on the low-end. If you can fit your data onto a single 2TB disk, then it's much more reliable than tape, and cheaper too.
Tape is also a lot less reliable outside data centres, because practically nobody designs those drives to survive the dust in a typical office environment. I've seen about 50% or more of the tape systems out in the field fail at least once a year, but I've never seen a disk based backup system fail. Note once, not ever.
Also, tapes aren't as robust as people think they are. One of the hardest IT troubleshooting jobs I've ever worked on was a tape system that had regular failures. Turned out that the IT guy liked to throw the tapes up in the air (only about a foot or two), and then catch them. He didn't drop them or anything, but that was enough to cause regular backup failures. That's less robust than a (powered off) disk drive.
I've got a feeling that tape reliability numbers are massively exaggerated in marketing materials. For example, I once had a tape getting repeatedly loaded, read a little bit, and then unloaded overnight because of a software bug. It was destroyed. Think grooves etched into the plastic casing, and the tape worn to the point of transparency. That got me thinking, and I looked up the numbers.
People get confused by numbers like "1,000,000 passes" in the specifications. You have to read the full version: "1,000,000 passes on any area of tape, equates to over 20,000 end to end passes/260 full tape backups". People forget that LTO makes many passes per backup, so suddenly you're down to a three digit number of backups instead of the huge sounding million they start with in the brochure. Throw in verify-after-backup, and it's only 130. If you back up daily, that's just over four months before your tapes are worn out, according to the spec. Meanwhile, even consumer grade hard drives can last for years, even decades.
Re:To Tape... (Score:4, Insightful)
Multiple backups to multiple media.
More than once I've seen tape recovery atempts fail because a streamer didn't work correctly, the tapes had manufacturing faults or their software of choice was nothing but a useless piece of shit.
Re:To Tape... (Score:4, Informative)
Yup; and note that "backup" these days often includes mirroring on other live system.
I've seen backups to tapes and external disks fail more often than I've seen them succeed. I've also over the years seen any number of estimates that around 50% of backup tapes aren't usable when they're needed. I've often been suspicious that there may be large error bars around that number, but so far I haven't seen any evidence on the topic. This makes me further suspicious that the people writing on the topic are assuming that I'm too stupid to understand anything more detailed or informative than "50%", implying to me that the reality could well be even worse than that.
OTOH, since large (whatever that means) disks became reasonably cheap and network access became inexpensive, around 10 years ago, it has become feasible to use rsync for backup. It's easy enough to install more disk than you need on several widely-separated machines, and have all of them back up all the others. This is especially easy if they happen to share a lot of the stuff that needs backing up. The data is stored in its original form, and if your backup media dies, you know about it very quickly. Restoring always works, because it's done via another rsync, which you know how to use (and is documented all over ;-).
The only real "gotcha" I've seen with this came in working on projects that had a mixture of Macs and Windows systems, together with the usual unix/linux-based servers. The hokey attempts at "caseless" file names on OS X and Windows, not quite the same on the two OSs (or on different releases ;-) causes some, uh, "interesting" name collisions and loss of files. The unix/linux systems can back up OS X and Windows files without problems, since they are agnostic about charsets, but if you try to back up their file systems on an OS X or Windows box, you tend to lose files due to multiple names being mapped to a single name.
I've actually made this work, by learning enough about the OS X and Windows file-name munging to avoid the problem. But we haven't (yet) learned how to avoid the problems with OS X and Windows attempt to mirror each others' file systems. So our rule amounts to "Do all rsyncs from OS X or Windows to/from the unix/linux servers, and don't let OS X or Windows mirror a file system from a different species of OS." (If you must do this, test the results thoroughly to make sure that neither OS is mapping two file names to the same byte string, and repeat the tests frequently to verify that new files don't trigger the problem.)
(I can already hear the shouts of "Mirroring isn't backup, you idiot!". I fully intend to ignore such comments. The term "backup" means any sort of copy that we can later use to restore our data. It doesn't mean tapes. It means anything that I can restore my data from. Nothing more or less than that. So whatever you're trying to sell - or prevent me from buying - is irrelevant. All I want to know is how good your favorite product is at keeping and restoring my data. Well, OK, I also want to know whether I can keep my data private, so I'm ignoring all "cloud" backup schemes. But data security is a different topic from data safety, so I'll ignore that here, too. ;-)
Re:To Tape... (Score:5, Insightful)
well, doh. do it to tape once a year. it's unlikely all hd's would fail at the same time.
checking _what_ was backed up on the other hand.. how many companies check that..
Re: (Score:3)
Flash drives? Flash drives can fail in ways that make the data absolutely unrecoverable, they have a relatively low capacity and are expensive at any meaningful capacity. Better than tape, maybe, but better than hard drives?
Re: (Score:3)
Most industries don't need that kind of historical data. What kind of data do you need from 5 years ago that isn't on your most recent backups?
Ah, yes, that was often a common sentiment before the Sarbanes-Oxley Act of 2002.
My own backups (Score:2, Interesting)
I run backups at least once per week, and if I make major changes to my system (I self web host), I backup several times per week. This week I lost a 500GB drive. I used ddrescue to recover all but 19MB, although after the filesystem was restored, I lost more than 19MB. Backups saved me huge amounts of time and trouble. The system was down for nearly three days. Data recovery took time, getting a new drive and installing an OS on it, and then rebuilding the system. Its been back for two days. I don't
Re: (Score:2)
Anecdotal evidence: I personally just installed ubuntu using software raid 1 on a desktop system - it added a single step to the installation process, and required me to use the "alternate install" cd instead of the standard live cd. The server install already has the required files.
Easy-peasy, although raid isn't a substitute for backups.
Re: (Score:2, Informative)
RAID is not a Backup
Re:My own backups (Score:4, Insightful)
The purpose of RAID is to avoid most of the cases where you would have had to restore from a backup, such as a drive failing.
Re: (Score:3)
Also afaict linux raid can support raid1 sets with more than two drives. So you can mirror onto a drive for backup purposes while still having two drives in the live system at all times.
No wonder they are switching to clouds (Score:2)
After one failure the costs alone will be realized. I personally never worked in a place that did not have a daily backup. Hell, I knew someone fired because he messed up a daily backup for 2 days but with no data loss just because if it *did* happen it would be catastrophic.
The only place I knew who did it weekly was a small computer shop.
Are companies this cheap today run due to excessive cost accounting cutting and right sizing? I find this too hard to believe
Re: (Score:2, Insightful)
Re: (Score:2)
Re: (Score:2)
"And, no. I don't know why they don't get fired for telling such a poor lie."
What's your data set size? What is their budget? If you have 5 pentabytes of data needed to be backed up, and their hardware budget is $4.50 (four fitty, I say!), then that is not a lie.
I'm going to assume that your data size is reasonable and that they do have an adequate budget. (Just say so next time)
Re: (Score:3, Interesting)
Surely it completely depends on the size of the company and the importance of the data?
Losing a week's work is *NOTHING* for a small business holder if they have all the normal records in order (it might only take an hour to bring them back up to date), but having a daily backup might actually be a huge chore for them if it involves swapping disks, wearing out tapes, has to be manually initiated, the servers are only on during the working day, etc. Losing a day's work for a 2000-employee company may be a b
Re:No wonder they are switching to clouds (Score:5, Insightful)
I hope you're a troll, because if you're not, this is just about the most insanely fucked up attitude to data protection I've ever seen.
A blind "Let's just backup the whole server" isn't an effective backup strategy ...
If I was your boss I'd fire you on the spot for being that monumentally stupid. The cost ratio of an additional ~20GB of tape compared the enormous cost to the business if the server can't be restored quickly and reliably is staggering. Think $1 saved while risking millions of dollars of potential lost work! (1.6TB LTO5 tape = $80)
a) "what if you upgrade to a larger disk" -- there is no backup system on Earth that can't restore to bigger disks. Most systems can restore to smaller disks too.
b) "what if the server is slower than others and people start moving their data to something" -- you back up both of them. Restoring too much is almost never as bad as restoring too little.
c) "just blanket-backing-up is likely to lead to problems later on" -- no, it doesn't. Achieving 100% coverage ensures that no matter where you data was on a server (or which server), it's on a tape somewhere.
d) "notifying IT of new things that need to be backed up" -- have you met humans? This never happens reliably, and can't ever be made reliable.
Back in the real world, a 100% complete backup of a typical Windows server can be restored without knowing the password, to dissimilar hardware (even virtual machines), and without needing the "original install disks". When it's done, it'll boot up, maybe reboot once or twice to fix up its drivers, and then your server is back, working as it did before. Compare that to a "data only" or partial backup. Now suddenly you're chasing down design documentation, passwords, IP addresses, software, serial numbers, and you haven't even started to restore anything yet. The clock is ticking, and the customer is breathing down your neck.
A week's work should take no more than two weeks at ABSOLUTE maximum to recreate
Recreate from what? Memory? Including data that was 100% electronic, and never seen by a human? How do you recreate your emails? How do you recreate your audit logs? How do you type back in non-textual data like digital images or audio recordings? How would you even know what's missing?
I'd rather have a decent monthly, than an imperfect daily
That's a false dichotomy. The total data stored is the same, you're just altering the frequency. The same amount of storage is needed, the same bandwidth is needed, and it ends up costing the same.
I'd rather have three backups a day than monthly backups. Losing a day of work could mean a contract fails to go through. I've been in a position twice now where users have come to me literally crying and begging to retrieve a document they only started working on that morning that they had deleted accidentally... at 8pm, minutes before a deadline for a multi million dollar deal. After experiences like those I've often set up incremental backup frequencies as rapid as 15 minutes.
So lets recap... your sum total DR experience is you once walked into a poorly supported environment, and gave them some even worse advice, without ever being in a position to be responsible for an actual real world recovery.
Well, take some advice from someone who's restored terabytes of data, and was responsible for the protection of over a petabyte spread across thousands of servers at over a dozen organisations:
#1 There are no time machines -- you cannot go back in time to fix a mistake in your backup strategy after a disaster. It's too late. You've fucked up, it's your fault, and you can never, ever, fix it.
#2 Back up everything -- I love genius IT folk who like to shave 1% off their backup times by excluding those useless temp and log files, also excluding 'useless junk' like their database transaction logs in the process.
Re: (Score:3)
I'm not sure if you've missed the point of the post entirely, which was stated in the first sentence, or got into a rant and went back to rantify it more afterwards. If you are a multi-million dollar establishment, I'm not suggesting you SHOULDN'T do dailies or anything else - because the cost of not outweighs the cost of. You're applying large enterprise IT to the world and that's exactly the point I make.
The drive to go with the tape you specify can cost more than some COMPLETE small business IT systems
Re: (Score:2)
We're a small business. I rotate 10 offsite HDDs. I only rotate offsite once a week. Each day I take backups of files between servers, so as long as several servers don't get taken out at once then things are fine. If anything happened that took out all the servers at once, I think we'd have bigger issues to deal with than losing up to 5 days of design work.
Our file server has volume shadow copy for restoring previous versions of files, and our databases are backed up each day onto the file server. So if pe
Re: (Score:2)
Tape isn't slow - at least not real tape drives. In fact, if you're looking an LTO4 or LTO5, the tape is probably faster than the disk subsystem in your server and you'll do the mechanism damage by not feeding it data fast enough (shoe-shining).
Re: (Score:2)
Easy (Score:5, Insightful)
Re: (Score:2)
Also, it'll never happen to them. And it doesn't make them any money.
Re:Easy (Score:4, Insightful)
Document the hell out of that!
See if you can cause a server to be down. When the CEO screams tell him such and such wanted to save $400 and didn't think it was worth it save $40,000 an hour on lost productivity to save $400 now. Sorry that had to happen during that $ 1,000,000 deal you lost. Heads would roll faster thanyou can say fired. ... unfortunately, this makes a cloud all the more attractive that could end your own job :-(
Companies are really stupid today! Normally I do not use exclamation marks, but the latest scaffle in Thailand where every single hard drive is made on earth in an all your eggs in one basket to save on 5% economies of scales cost, to companies producing junk, and now this drives me crazy. It bothers me because so many of us are out of work and these companies are penny wise but certainly dollar dumb. To me no tape backup in any fortune 2,000 company is the equilivent of having no fire insurance. A single disaster could cost you everything and it is so cheap when you rake in millions.
risk (Score:5, Insightful)
Most companies have no risk management, and no clear picture of the risks their business faces.
The result of "intuitive" risk-non-management is that the usual human flaws have full impact. Basically, aside from a narrow middle ground, all risks are wrongly estimated.
Computers are too reliable (Score:5, Interesting)
The result is that people just expect them to work flawlessly -- not to fail. They also ignore other risks. I put in a machine at some customers a couple of years ago. They did not want to pay extra for backups -- ''yes, we will do that later''. They knew that I configured mirrored (RAID 1) disks. I set up a backup from one part of the disk to another and reminded them every couple of months that they would loose everything if it was destroyed or stolen.
Then a few weeks ago another unit on the industrial was torched -- arson. I have finally pursuaded them ... I am putting in another machine on the far side of their factory that will take a daily rsync, and USB plugin disks that they will backup to weekly & take off site.
These guys are not stupid in what they do professionally, they have an annual turnover of more that £1 million. Why does it take a fire at a neighbour to make them see sense ?
Re:Computers are too reliable (Score:5, Funny)
Then a few weeks ago another unit on the industrial was torched -- arson. I have finally pursuaded them ...
Ahhh yes, the old "Nice data you have there, be a shame if something happened to it" trick.
No need to explain, friend, we all know what happened there.
I am curious if you did the deed yourself or contracted it out to a professional, though. ;)
Re: (Score:3)
I am curious if you did the deed yourself or contracted it out to a professional, though. ;)
He probably suggested Sony laptop batteries. They took care of the rest themselves.
The Benefit of Backups is Non-Obvious (Score:5, Interesting)
Seriously. Most people aren't willing to put in the effort to determine just how bad things would be if they had to resort to their (often inadequate) backups, and therefore they aren't willing to pay the time and capital to get adequate backups.
If you want your company to get better backups, run a simulation of what would happen if something failed. What's the best recovery you could do? What business would you lose? Then calculate the probability of that failure occurring, and be generous.
Re: (Score:2)
If you want your company to get better backups, run a simulation of what would happen if something failed.
Quickest and easiest way to do this in a dramatic way?
During a chat with the CEO, reach over and shut down a random server. Ask him where all the datas went, and how the company is going to operate without whatever data was on that server for however long it would take to recreate the data. Might get you fired, but might also make him reconsider the backup project he just axed because it would require him to put off that new car purchase for 6 weeks.
Re: (Score:2)
Unfortunately, the penny-savers are also experts at the blame game, and have a disaster recovery plan of "point the finger at the IT guy".
Make sure you document every discussion of backup solutions with the beancounters, in case something actually happens before you convince them not to be dumb.
Re: (Score:3)
If they deny, ask them why they have a fire insurance.
backup often, and respect the 'rm' (Score:5, Insightful)
especially when combined with 'find' and 'xargs', in what is supposed to be a simple task.
If you don't, you'll do something like what i just did ("worst typo in a decade"): you see, i was trying to update emacs and wanted to purge all the .elc files from ~/.emacs.d
Unfortunately, through a bad typo, some miss-applied keyboard shortcuts, and rushing through without mounting a scratch monkey... what actually ran was effectively "find ~/.emacs.d | xargs rm".
accidently deleted the 'grep'. Oops. 15+ years of elisp/etc destroyed.
Was it backed up? Nope! Been meaning to check it all into git, but always put it off as a "minor, unimportant" task I'd get to later. Of course, we all think that way up until the disaster hits...
*sigh*
Wouldn't have happened if you used vi (Score:5, Funny)
Re: (Score:3)
I may not have a proper backup of 100% of the source code I have ever written, but I simply cannot imagine ever not having copies of the stuff written before my last big system upgrade (unless a fire takes out everything.)
It is simply unthinkable to not have a copy of at least 7 of those 15 years.. never in 15 years was a copy made.. seriously?
This is beyond "backup" territory, because even people that dont "do" backups have burned their shit at
Re: (Score:2)
Accidentally deleting things, is usually fixable, as long as you immediately stop writing new things to that filesystem. Search for undelete tools for whatever platform and filesystem you use.
Or run windows and take VSS snapshots of the shares where users store their data. We haven't had to restore from tape/disk data that someone accidentally deleted in years - the users just restore it themselves from VSS.
Re: (Score:2)
ext4, which was somewhat light on tools last i checked (have to look into that...)
More to the point, though... halting writes wasn't really possible, due to a lot of unrelated (and probably more important) things thrashing the disk.
It's a beautify example of why a well though-through backup plan is important, unfortunately. RAID doesn't protect against being an idiot with 'rm', and it's probably a good idea to research things like those undelete tools before you need them...
Re: (Score:2)
I don't know about Linux or Mac but Windows does not delete to the recycle bin either when you delete files from the command line.
Re: (Score:2)
Is there a linux recycle bin implementation?
Yes, it's called "Trash" - but it only works if the delete function uses it.
Tape is frightening in and of itself (Score:2)
I think my record for getting things restored off tape is about 10%. It mostly seems to be a placebo.
'Do we have backups of FOO?' 'Oh sure, we backup everything nightly.' 'Thanks, could you get me... FOO/BAR?' 'Sure. Justasec.'
Two hours later I get the call. 'Uh, we're having some issues here, it'll be a bit longer...' Two hours after that 'How badly did you need FOO/BAR?'
Either the automatic backup system was failing early on in the backup and had been for months and nobody had noticed the error condition,
Re: (Score:2)
Re: (Score:2)
To be fair, he wasn't doing the backups. Or is he supposed to take over the IT section while he is at it?
Re: (Score:2)
I'm not going to blame myself - it's not MY system, which works great; tape is the IT department's system. And that's the problem. The whole tape 'ecosystem' seems to be too fragile for most IT departments to handle. If you get great IT guys I'm sure they can run a fine tape backup system, but usually they know even less about computers than the other employees.
Mirror to disk is much harder to screw up (though certainly possible!) and easier to verify. Both places I've been that had disk backups had no pro
Re:Tape is frightening in and of itself (Score:4, Informative)
I have had SDLT tapes I recovered from the floor of a server room after sitting for 3 days under water that read just fine in a drive after being dried out.
If you get a decent tape backup it has built in redundancy, 48 tracks of data with CRC checking and redundant data written in case that CRC fails.
Granted, I'd not rely on them surviving kicked around the floor, but a real tape backup system is far, far more robust than the under $1000 garbage sold at newegg.
Re: (Score:3)
at least 25 times, SDLT tapes are durable as hell.
The reason, as always, is induction (Score:2)
Yeah, induction [wikipedia.org], folks.
It's been 6 months since I began my startup. That's 3600x24x30.5x6 seconds, or 15,811,200. 15,811,200 seconds of no computer problems, and 0 with.
Extrapolating out, that means infinite problem-free computin ??#@ NO CARRIER
Offsite backups (Score:2)
Offsite backups are what a lot of companies don't do. They might back up to tape, but the tapes are stored in a pile next to the server. And they never test them.
Re:Offsite backups (Score:5, Insightful)
Offsite backups are what a lot of companies don't do. They might back up to tape, but the tapes are stored in a pile next to the server. And they never test them.
Agreed. One of the many phrases in my litany on customers backing up is "Yes, that safe is fireproof. For paper. Plastic melts at a much lower temperature than paper burns." Not only do not enough companies run backups on a regular and timely basis, but too many of the ones that do run backups don't see the need for having the storage media offsite.
Don't have the bandwidth for a cloud backup (Score:3)
I'm not sure how everyone gets so ecstatic about those cloud backups. When we would need to send all our data over the internet connection it would take an unworkable amount of time to complete the backup.
Even to the local LTO-4 drive, which runs at over a gigabit per second, the backup takes an appreciable amount of time.
Cloud backup may be good for a 3-man company doing document editing, but with the amounts of data that are common these days, and the speeds of internet connection that you normally have, I don't see it as a realistic possibility.
Re:Don't have the bandwidth for a cloud backup (Score:4, Funny)
Oh shush, everyone knows that the Cloud will cure all that ails you! Storage, backup, canker sores? The CLOUD!
Re: (Score:2)
Just taking the devil's advocate position, here, but it sounds as if you have enough data that it might be worthwhile to move your data storage into the cloud, and access it from there - let the remote site's IT guys worry about the backups.
Slashdot (Score:5, Interesting)
I've always wondered if /. bothers to back up the stories and posts...
Anyone here know?
How about for other 'trivial' sites, like reddit, youtube, etc?
Re: (Score:2)
when the operation gets big enough, it's just simpler to have data redundancy by other means.
Re: (Score:2)
They are backed up in the Google cache.
DATA (not disks, or tapes, or storage) (Score:2)
You need to think about what data you are protecting, why, how you RESTORE it, and whether it needs an offsite solution at all.
If you do that properly you knock a zero or two off your technical volumes, every time.
Conversely, it's seldom worth doing that at $SMALLCO because the entire enterprise fits into a Terabyte or less and storage is cheap. The danger is that throwing it onto something offsite - whether it's disk, tape, cloud, or ferromagnetic core memory - can lead to lazy thinking about what happens
Hard drives have replaced tape... (Score:2)
... because hard drive costs have come down so much it's just cheaper to buy a bunch of hard drives and mirror like crazy. Another factor is speed. Backup up costs time and time is money. So it makes sense that more and more organizations have moved to to mirroring/RAID solutions.
RAID is pretty damn robust these days also and with drives as cheap as they are you can create many mirrors at once. I've owned raid 5 over the past 7 years and I've never lost data drives have always died 1 at a time on a disk
Re: (Score:2)
Wait until your boss deletes that important document and your RAID system has deleted it on all drives in the array immediately at his request.
Or your business application is slowly corrupting the database and it is noticed (or finally confirmed) only after 3 weeks of use.
At that time you want to be able to get old data back. This is not something your array is going to be able to provide you.
Re: (Score:3)
"Wait until your boss deletes that important document and your RAID system has deleted it on all drives in the array immediately at his request."
Do you know how RAID works? it stripes the data across ALL drives. if you think RAID5 keeps separate copies you are a little nutty.
RAID 1 is mirroring, but nobody at all uses it other than home users that are paranoid and like paying 2X for their storage space.
and to stop idiot bosses, you put in place a RCS. revision control does wonder to solve the "oops I de
Re: (Score:2)
Sure, if you only have like 1-10TB harddisks are fine...
But how to do you handle backups of say 50TB of data every week? Awful lot of disks to swap, copy times will be so long you'll need to start the next weekly while the last job is still running etc etc.
Tape still has it's place. It's not the universal solution for all backup problems, but for large datasets it's still king.
Re: (Score:2)
RAID won't save you from the ohnosecond when you realise that you actually typed "rm -rf / tmp/*" though.
Re: (Score:2)
RAID is no substitute for backups. Yes, RAID5 will handle a single disk crashing. ... but if the building burns down, "where did my datas go?"
if more than one drive fails simultaneously...
if a drive dies while you're still resyncing your array from the first disk crash...
if you don't notice you have a failed drive because you didn't install the disk-monitoring tools...
You can mirror all you want to, and if the data never leaves the room, it's not backed up.
I guess a potential solution using only RAID would
Re: (Score:3)
"RAID is no substitute for backups. Yes, RAID5 will handle a single disk crashing."
You know how many times I have seen a raid 5 go down hard with a single drive failure? too many to count.
Raid 5 lost a drive, no problem the hot spare will swap in and the rebuild starts..... and then drive #2 fails..... all the data is now utterly hosed as it failed during a rebuild, or better yet, the freaking hot spare was an old dead drive that someone failed to pull out of the cage during the 2 years when they cruise
Complacency (Score:5, Insightful)
To most PHBs a computer is a toaster. That's right, an appliance. Nothing more. Most have no idea of the nursemaiding a computer needs, or how vulnerable company data is when it's reduced to 0s and 1s. Until the "unthinkable" happens and the toaster breaks.
Now. I used to be in the biz of disaster recovery. It was lucrative when consumers who valued their potentially lost data found me and asked me "Can you do it?" and I said "Let's have a look". My success rate was something like 40%. Surprisingly (for me anyway), my better corner was recovering data from physically damaged flash drives. Out of several dozen of those I only had one that I couldn't recover anything from. Hard disks are a different animal, and the whole thing can be frustrating when you sit back for a minute or five and consider how much easier your work would be (and how much hardware you can sell) if your client had the benefit of hindsight and someone around who knew the shit of which he spoke - so a week-long recovery project that may or more than likely may not work turns into a two hour exercise in restoring on new hardware and sending the happy client on his way. Now, that is an exercise in getting repeat business through recommendation.
As far as "accidental" deletion of data: there is only so much you can do to protect the stupid from themselves - on a network share, for instance, you can deny users the right to delete files. Job done. In an environment that is supposed to be secure, that's a good start. VSC is another handy tool but you don't need to tell the stupid that their fuckups are (sort of) covered - it breeds complacency and does little to nothing to train responsibility.
Only the idiots are. (Score:3)
I backup to backup server every hour and then spool to tape every night.
Once a week that weeks tapes get sent offsite and old 3 month old tapes are returned for re-use.
If the company's IT department is NOT doing daily backups then it's IT department is ran by an idiot.
If management will not pay for it, then they need to be told, "so everything you do today does not matter and has no value? because that is what you are saying when you say backups are not important. Hard drives are not reliable, if they crash tonight and we lose everything, how much money do we lose?"
You have to talk to them in management speak... AKA money.
why can't I get my mom to back up? (Score:3)
I expect a lot of companies are just like my mom. having never experienced data loss, she doesn't see the point of backup.
Not true... (Score:3)
>showing 10 percent were only backing up to tape once per year
You mean showing 10 percent were only backing up to tape once per year to a paying third party vendor....
instead of in house which is cheaper when you know what you are doing...
I love when they twist these headlines to make them more attractive..
You Can't Take Credit For Avoiding A Disaster (Score:3)
Pretend a guy prevents the 9/11 attacks by requiring all airliners to have a bulletproof, locked cockpit door. Hundreds of millions of dollars are spent, and prices go up. Everyone complains about government regulation. The attacks never happen. The instinct was that we just wasted a bunch of money. We spent hundreds of millions on H1N1 vaccine for an outbreak that never happened. But if it did, the vaccine would have saved many lives and prevented great financial loss. But it never happens, so it's seen as lost money.
You push for backups, and you spend tens of thousands on it for years. There is a disaster eventually that you recover from quickly, but perhaps you aren't around to take credit for it. Or you restore and everyone thinks that it was supposed to because we spent so much on it. No one will think, "Well, we just saved millions of dollars of downtime because of the backup." The thinking is just, "Oh, well it was supposed to. We spent so much on it."
This kind of myopia is commonplace in the world. We can invest money to treat everyone for chronic illnesses such as diabetes, high blood pressure, and high cholesterol with really cheap drugs. We can vaccinate everyone from childhood diseases. "Oh, but vaccines cause autism." Yes, but your child isn't paralyzed from polio. The avoided disaster is never quantified; only the cost spent shows up on your calculations.
Re:Who uses tape any more? (Score:5, Insightful)
RAID is not a replacement for a backup.
RAID will safeguard you against the failure of a single disk (if and only if you monitor the system and replace disks as they fail), but backup will give you back your data as it was before your application destroyed it or your user deleted it. That is something completely different.
Re: (Score:2)
You're so wrong ...
Tape is the ONLY way to make serious backups and do archiving once you start having serious data volumes. 50TB of data is a LOT of disks... now do daily increments, weekly full backups with 2 months retention... also single disks are always slower than streaming to LTO4 or LTO5 tapes so your backup window becomes too big to handle.
Re: (Score:3)
There's no need for a robot when all the "tapes" (aka HDDs) are all accessible while stored: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ [backblaze.com]
(IIRC this was on Slashdot before).
But if an org only needs 25 or similar magnitude of drives you can go Dell (e.g. stuff like MD1000 or MD1220) for a many times the price/capacity. Most orgs with those needs can afford it.
If I were running my own company I'd go with Dell/etc and HDDs first, but if my required storage capacity curv
Re: (Score:2)
Or at least don't answer them three sentences later.
It's been a while since you wrote a college paper, huh?
Re: (Score:2)
Of course when someone had deleted half of the data in your finance database and you would notice it on the year or month close, you would already have overwritten all your backup media and lost your data forever.
Some time, someone will come at your desk and ask "I'm sure I had that contract in my documents folder last year but when I look now it has vanished".
At that moment you will realize that your backup rotation scheme is not as clever as you first believed.
Re: (Score:2)
Encryption?