Forgot your password?
typodupeerror
Security IT

Why Do Companies Backup So Infrequently? 403

Posted by samzenpus
from the save-your-work dept.
Orome1 writes "Businesses are on average backing up to tape once a month, with one alarming statistic showing 10 percent were only backing up to tape once per year, according to a survey by Vanson Bourne. Although cloud backup solutions are becoming more common, still the majority of companies will do their backups in-house. Sometimes they will have dedicated IT staff to run them, but usually it's done in-house because they have always done it like that, and they have confidence in their own security and safekeeping of data."
This discussion has been archived. No new comments can be posted.

Why Do Companies Backup So Infrequently?

Comments Filter:
  • To Tape... (Score:5, Insightful)

    by sensationull (889870) on Friday November 18, 2011 @04:24AM (#38095526)

    Portable HD is cheaper and faster, even for stacks of them. Small businesses may be using a bunch of these in place of tapes.

    • Re:To Tape... (Score:5, Informative)

      by adolf (21054) <flodadolf@gmail.com> on Friday November 18, 2011 @05:22AM (#38095818) Journal

      Indeed. My own [small place of business] does its backups this way with Acronis:

      -
      Every day, a drive is plugged in.

      Every evening, an incremental backup is performed on that drive that is consistent with whatever the backup drive already had on it. (It just takes a few minutes for Acronis to get this done.)

      After that it is removed and taken off-site. A different drive is plugged in the following day.

      Fridays are special, in that the drive gets automagically erased, and a full backup is performed mid-day. But then the evening incremental ritual is the same as any other day of the week.
      -

      Losing a backup disk is, at most, 24 hours worth of loss, and only then if it is coincident with losing the main system.

      FWIW: We use the cheapest 2.5" laptop drives available, in the cheapest bus-powered 2.5" USB enclosures we can get our hands on (I think the last round of them cost us $5, each). And we test our backups randomly, whenever the accountant decides he wants to see what things looked like last week/last year. Acronis then cheerfully does a bare-metal restore from [random backup drive] onto our spare server-box, with 100% success.

      Now: We're not a media company. We don't have tens of TB of changing data to back up on a daily basis. But most other small(ish) companies don't either...

      Who needs tapes?

      • by Splab (574204)

        That doesn't make sense, if you are using a new drive each day, how does the friday wipe come into play? Where do you store your thousands of drives? And how the hell do you manage to pick up drives for $5?

        If you are not using a new drive each day, how do you recall a backup from a year past when a drive is wiped every friday?

        • Re:To Tape... (Score:5, Informative)

          by cdrudge (68377) on Friday November 18, 2011 @09:30AM (#38097216) Homepage

          There are many different backup strategies depending on what you are backing up and if your backups are designed for disaster recovery or archival of data.

          Making some presumptions here to answer your questions:

          They use a rotation of 5 or (7 drives if including weekends). Monday's drive is always the same, as is Tuesdays, Wednesday, etc. Each drive is kept offsite and only that day's drive is brought onsite while others remain offsite, or are taken back offsite.

          If on Thursday the server goes down, they just request Wednesday's drive for the restore. If that backup was unsuccessful or the drive just happened to fail, then they lose Wednesday's business and go back to Tuesdays. Repeat until they get a successful restore.

          Fridays basically are not an incremental backup. Well, the evening is, but the midday backup is a full backup and then is incrementally updated from the 1/2 day of business that night.

          With this scheme, they can't restore a backup from a year ago. Depending on what is being backed up, they may not have to go back a year. Accounting information for instance from a year ago may be in the current backup set. If you are looking to restore something that was corrupted or deleted at an unknown point and wouldn't be in the current backup, then it would just take 1 disk out of the rotation once a month for instance and replace it with a new drive. A previous company I worked for recommend this strategy with point of sale systems we sold. We gave them a box of 10 backup tapes. 6 nightly backup tapes and 4 weekly tapes. The weekly tapes were in a 4-week rotation while the nightly were in a 1 week rotation. If they wanted to do a monthly rotation they could purchase additional tapes.

          The $5 wasn't for the drive, it was for the enclosure and/or drive adapter.

      • Re:To Tape... (Score:4, Insightful)

        by GameboyRMH (1153867) <gameboyrmh@gmail.cCOBOLom minus language> on Friday November 18, 2011 @10:49AM (#38098054) Journal

        FWIW: We use the cheapest 2.5" laptop drives available, in the cheapest bus-powered 2.5" USB enclosures we can get our hands on (I think the last round of them cost us $5, each).

        Did you look at using trayless hot-swappable drive racks? They're very handy, you just plug the bare drive in and away you go at full SATA speed.

    • Re:To Tape... (Score:5, Insightful)

      by CmdrPony (2505686) on Friday November 18, 2011 @05:23AM (#38095822)
      As much as Slashdot crowd hates anything cloud, I think cloud based backups for small businesses would be perfect. They just have to make sure they get one with backup guarantee and SLA (ie., don't just get Dropbox - get actual quality one that tailors to such services).
      • Re:To Tape... (Score:5, Informative)

        by aix tom (902140) on Friday November 18, 2011 @05:36AM (#38095906)

        Perhaps for *very* small businesses with not much data, or where the restore time is not a critical factor.

        Even with our 32Mbit connection at work it would take about 40 hours in transfer time alone to get our data "out of the cloud" in a disaster scenario. (And part of that bandwidth would still be needed to keep other operations running, so it might even take longer) You can perhaps make small, incremental backups into the cloud, but when you need the data in an emergency you need it as fast as possible. Restore from tape took two hours. (We had a chance to do it "for real" earlier this year ;-P )

        • Re:To Tape... (Score:4, Interesting)

          by Eivind (15695) <eivindorama@gmail.com> on Friday November 18, 2011 @06:11AM (#38096048) Homepage

          But you can combine:

          Make local backup to disc, and use cloud-based backup as the off-site-storage.

          This was you easily get near-zero-administration. You get quick restores for the more common scenarios (server died, user accidentally deleted something important, disc crashed).

          Yes you get slow restores for the uncommon cases (those where both the primary storage, and the on-site-backup are fubared) this happens if, for example, a fire destroys the building or thieves steal everything that looks electronic and expensive - but in these scenarios it's going to take some time to get back on the air anyway, aslong as you ain't got atleast two physically disjoint datacenters.

          If you ain't got huge amounts of data, this is cheap. If you *do* have huge amounts of data, it can get expensive and inacceptibly slow, in that case you might need a different solution. We've got around a terabyte in the cloud (it automatically de-duplicates, so the same file is only ever stored once), and this costs us $750/year which is cheap enough to be a no-brainer. (it'd be different if we needed to backup hundreds of terabytes though)

          We've got a 100Mbit link, so restores happen reasonably fast, but a complete restore from nothing of everything, would still take time.

      • Re:To Tape... (Score:5, Informative)

        by sirlark (1676276) on Friday November 18, 2011 @06:27AM (#38096134)
        Or for small businesses in the developing world, where you pay $100 a month for 4Mbit/s (South Africa) and the fastest connection is 10Mbit/s but only available with usage based billing, stick to tapes or hard drives. The cloud is just too damn slow, or too damn expensive.
    • Yes, but I'm not interested in cheaper and faster. I want lightweight portability, reliability, and drop-ability. When I throw a few tapes in tupperware to be picked up I am fairly certain those tapes will survive the trip to and from the storage facility driven by who-knows-where-they-found-this-guy. I can effortlessly take a few home. HDD and SSD are too sensitive and too heavy. Sure they are easy to test because they are so fast and have huge capacity, and it is definitely time to move on, and I sho
    • by Manip (656104)
      You're mistaken, tape is a lot cheaper than hard drives. You can buy 1 TB of tape for as little as $30. The tape systems themselves are more expensive but the actual storage saving of using tapes very quickly makes it a worth while investment. I'd recommend anyone go and check the cost of tapes Vs. the cost of hard drives on Amazon if you care to.

      I am aware that Hard Drive prices have recently increased due to the flooding but even before that, back when HDD prices were low, tapes still under-cut them.
      • Re:To Tape... (Score:5, Interesting)

        by SuricouRaven (1897204) on Friday November 18, 2011 @05:44AM (#38095938)
        Hard drives have a high cost per gigabyte and no fixed cost. Tape has a low cost per gigabyte but a very substantial fixed cost - a high capacity tape drive cen easily set you back a few thousand pound. There is a crossover point: For any volume of data larger than that, tape is cheaper. For any volume lower, disk is cheaper. Few companies are on the tape-is-cheaper side.
      • by sjames (1099)

        Where? A quick google shows $130-$160 for the tape which is actually 500GB but MIGHT store 1TB if the data is compressible enough. For $130, I can get a 1-2TB hard drive that will store an ACTUAL 1-2TB of data, more if I use compression.

        • by rezalas (1227518)
          You should have performed a better google search, since I buy LTO-5 tapes (1.5TB uncompressed, 3TB compressed) for roughly $60 each. Included in that cost is pre-labeling based on my specifications and series numbers from previous orders, and two day shipping. Roughly 1/3 the cost of the tiny tapes you found, and under half the cost of the comparable hard drive. Tape offers many benefits that you don't have in a hard drive, including better drop resistance (moron resistant), and the ability to be physicaly
      • Re:To Tape... (Score:5, Insightful)

        by Eivind (15695) <eivindorama@gmail.com> on Friday November 18, 2011 @06:17AM (#38096078) Homepage

        The thing is though, tape *sucks*.

        The most common reason for needing a restore, is accidental deletion. With modern backup-systems using online disc, the user can in this case simple open his backup-client, find the file, and click "restore" - time elapsed 2 minutes, help needed, none.

        Tape means talking to IT, and wait for hours, at significant personnel-cost. Unless there's a fancy automatic tape-switching-robot kind of deal, but if there is, the price is no longer $30/TB.

        Yes you need off-site-backup in addition, and it's acceptable for that one, to have higher latency, so perhaps tape is okay for that.

        • by omglolbah (731566)

          Amusingly some local tech staff members at a certain norwegian gas processing plant do local hd backups of their network shares and such for this very reason... It takes so damn long to get a backup restored that they have another level of backups locally to avoid having to ASK the central IT for help...

          Ugh..

          • So a memory hierarchy [wikipedia.org] emerged: local use on solid-state or disk, first-level backups on disk, and second-level backups on tape. That sounds reasonable for organizations big enough to need tape.
            • by rezalas (1227518)
              Organizations big enough to use tape have been using hybrid disk / tape backup solutions for a very long time. We use locally stored centralized backups that we can push to remote sites from disk, and then also perform offsite backups nightly with tape. It is really the only way to go when you have multi-terabyte incremental backups.
        • Re:To Tape... (Score:5, Insightful)

          by Blue23 (197186) on Friday November 18, 2011 @09:14AM (#38097084) Homepage

          The thing is though, tape *sucks*.

          The most common reason for needing a restore, is accidental deletion. With modern backup-systems using online disc, the user can in this case simple open his backup-client, find the file, and click "restore" - time elapsed 2 minutes, help needed, none.

          Tape means talking to IT, and wait for hours, at significant personnel-cost. Unless there's a fancy automatic tape-switching-robot kind of deal, but if there is, the price is no longer $30/TB.

          Yes you need off-site-backup in addition, and it's acceptable for that one, to have higher latency, so perhaps tape is okay for that.

          What you describe sounds like a tape-per-server solution, but anyone using virtualization or even having a reasonable size will use an enterprise backup solution that's user friendly regardless if the back end is disk, tape, or both. We use TSM to tape libraries of Ultrium5 drives (and disk pools for a backup landing place/1st day restore). Need to restore files or directories? Open up the local web client, click what you want in the GUI (and potentially a point in time you want to restore it from), and it's automagically done in 1-2 minutes. Regardless if ti's last night's backup or something from 3 years ago. Most of that time is the robot loading the tape(s) and seek time - getting data off the tapes is fast.

          We back up about 5TB of changes daily, and reclaim about 3.5TB of old no-longer needed data daily (much needs to be kept for extended periods for compliance issues), with over 2PB in-use storage. Power/cooling costs for tape barely move the needle, that much disk would have a large impact on our datacenters even using deduplication.

          Also, we have both on-site (for speed of restore) and off-site copies of everything. Easy to do, the tapes backed up at night make copies automatically during the day. This isn't a replacement for disk replication to alternate sites for site disaster, it's for normal restores and keeping everything we need for compliance reasons safe.

          I'm not saying disk is bad - disk is good. But tape is good too, and it's a different tool - fits a different niche than disk. Hammer and screwdriver will let you tackle more jobs then either one by itself.

        • Re:To Tape... (Score:4, Informative)

          by secret_squirrel_99 (530958) on Friday November 18, 2011 @10:19AM (#38097670) Homepage
          You're missing the point

          The most common reason for needing a restore, is accidental deletion.

          It is, but the most common use for tape is compliance, Many companies, and all public companies have compliance issues, SarBox, HIPAA, GLB, etc. Most require data retention of many years. In some medical settings as much as 21 years. Do you want to keep that all on spinning disks? or on CHEAP tape sitting in a box at Iron Mountain? In my environment, and that of many large companies, we no longer measure terrabytes, except at the individual database level. We manage several petabytes.

          disk based solutions are operationally ideal. Thats why products like Avamar and Data Domain do so well. But for large, long term, low access, storage, tape is still king.
      • by Lennie (16154)

        If you are using Amazon for backup you are doing it wrong. Amazon has a different purpose is much to expensive for that.

    • Re:To Tape... (Score:5, Interesting)

      by Anonymous Coward on Friday November 18, 2011 @07:38AM (#38096450)

      Disclosure: I work in backup and recovery for a living. I've been doing TSM [wikipedia.org] professionally for the past six years.

      So let's look at this idea that portable HD is cheaper and faster. I've managed backup systems that were backing up tens of terabytes every day. Let's say three 3 TB HDDs for the sake of discussion. Let's say five weeks' worth of data retention. That's 5*7*3 = 105 hard drives that need to be managed - call it 110, to take into account the fact that there are going to be a few going off and onsite at any given point in time.

      How reliable are those hard drives? How long until they fail? Bear in mind when considering your answer that hard drives are designed to be stuck inside a chassis that stays put (for the most part) - pulling them out of the system and lugging them halfway across town is not within their design goals. And then you need to worry about the cost of a hot-swap chassis so you can pull the drives out, and shove them back in, every day. Oh, and don't forget that most plugs have a limited life span - probably in the hundreds of swaps, maybe thousands (I don't know, and I'm happy to hear solid data about this.)

      And how long does it take to fill up a hard drive, anyway? Take the WD Green 3 TB as an example: 110 MB/s (source [techreport.com]), equals about 7.5 hours, best case. Sure, you can throw more hard drives at the problem in parallel, but that just exacerbates the whole question about reliability in transit. The Hitachi 3 TB is faster - 207 MB/s [techreport.com] - which takes about four hours to fill up. That's best case scenario, based upon the maximum data transfer rate - guaranteed it's going to slow down as the drive fills.

      Now consider tape. Consider wikipedia's information [wikipedia.org] on LTO when reading this. LTO4: 120 MB/s native, up to 300 MB/s compressed (two hours to fill). LTO5: 140 MB/s native, 350 MB/s compressed (three hours to fill). Pretty damn reliable in transit; they're designed to sit happily in their little plastic shells, in a box, and get thrown around (not quite, but they can certainly take more punishment than a hard drive can.) Capacity per cartridge: 800 GB/1.5 TB native (4/5); 1.6 TB/3 TB compressed. If you have money to burn, you could go for Oracle's T10000C drive: 5 TB native capacity, 240 MB/s native throughput (and that's before you get into the whole question of compression.)

      Now let's get onto the whole subject of financial data, Sarbanes-Oxley, and WORM media (so you know the data hasn't been altered since it was written out) ...

      Sure, tape sucks. It has major issues when data is scattered all over the place; mount time takes a while; and the drives need regular love and attention. But here's the thing - it survives today because it's better than the alternatives in its niches, and trust me, there's plenty of niches where tape fits in far better than the alternatives.

      If you only have a couple of TB of data to backup, the cost of setting up tape infrastructure will probably not be worth it. But when you're talking hundreds of TB - or better, petabytes of data (don't laugh, one client I did work for had over 2 PB of data in their tape library) - the cost equation swings over pretty damn fast. Tape is not dead. Far from it. I can't see the likes of IBM investing in developing LTO6 and LTO7 if there was no use for it. And why would Oracle sell a tape library that scales to 100,000 slots [oracle.com] if there's no demand for it? It's not about how to get the most bytes for your dollar - it's also about reliability, and that gets down to the usage. If I suggested portable hard drives to the clients I do work for, I'd be out of a job - because they simply won't cut it for their needs.

      • Re:To Tape... (Score:5, Interesting)

        by bertok (226922) on Friday November 18, 2011 @08:58AM (#38096950)

        I've had similar experience, but I've found that it swings back toward disk in a big way on the low-end. If you can fit your data onto a single 2TB disk, then it's much more reliable than tape, and cheaper too.

        Tape is also a lot less reliable outside data centres, because practically nobody designs those drives to survive the dust in a typical office environment. I've seen about 50% or more of the tape systems out in the field fail at least once a year, but I've never seen a disk based backup system fail. Note once, not ever.

        Also, tapes aren't as robust as people think they are. One of the hardest IT troubleshooting jobs I've ever worked on was a tape system that had regular failures. Turned out that the IT guy liked to throw the tapes up in the air (only about a foot or two), and then catch them. He didn't drop them or anything, but that was enough to cause regular backup failures. That's less robust than a (powered off) disk drive.

        I've got a feeling that tape reliability numbers are massively exaggerated in marketing materials. For example, I once had a tape getting repeatedly loaded, read a little bit, and then unloaded overnight because of a software bug. It was destroyed. Think grooves etched into the plastic casing, and the tape worn to the point of transparency. That got me thinking, and I looked up the numbers.

        People get confused by numbers like "1,000,000 passes" in the specifications. You have to read the full version: "1,000,000 passes on any area of tape, equates to over 20,000 end to end passes/260 full tape backups". People forget that LTO makes many passes per backup, so suddenly you're down to a three digit number of backups instead of the huge sounding million they start with in the brochure. Throw in verify-after-backup, and it's only 130. If you back up daily, that's just over four months before your tapes are worn out, according to the spec. Meanwhile, even consumer grade hard drives can last for years, even decades.

  • My own backups (Score:2, Interesting)

    by Anonymous Coward

    I run backups at least once per week, and if I make major changes to my system (I self web host), I backup several times per week. This week I lost a 500GB drive. I used ddrescue to recover all but 19MB, although after the filesystem was restored, I lost more than 19MB. Backups saved me huge amounts of time and trouble. The system was down for nearly three days. Data recovery took time, getting a new drive and installing an OS on it, and then rebuilding the system. Its been back for two days. I don't

  • After one failure the costs alone will be realized. I personally never worked in a place that did not have a daily backup. Hell, I knew someone fired because he messed up a daily backup for 2 days but with no data loss just because if it *did* happen it would be catastrophic.

    The only place I knew who did it weekly was a small computer shop.

    Are companies this cheap today run due to excessive cost accounting cutting and right sizing? I find this too hard to believe

    • Re: (Score:2, Insightful)

      by Belial6 (794905)
      Yes, they are. Our Sys Admins do backups for us, but they have decided that 3 days worth of rotating backups is sufficient. To be fair, after much gnashing of teeth, we got them to grudgingly up the amount to 7 days worth of rotating backups. Their claim is that disk space is too expensive for anything more. And, no. I don't know why they don't get fired for telling such a poor lie.
      • by igb (28052)
        Surely to God backup policies are not the responsibility of system administrators? They can propose, I guess, but the sign-off and the strategy must come from someone with a risk-management responsibility, and (in my experience in the UK) auditors won't sign off accounts without a discussion about IT resilience --- it's an "going concern" issue. If they are given a silly budget then ultimately there's nothing the Admins can do, but they should be banging at the door demanding sufficient budget and then te
      • by bryan1945 (301828)

        "And, no. I don't know why they don't get fired for telling such a poor lie."
        What's your data set size? What is their budget? If you have 5 pentabytes of data needed to be backed up, and their hardware budget is $4.50 (four fitty, I say!), then that is not a lie.
        I'm going to assume that your data size is reasonable and that they do have an adequate budget. (Just say so next time)

    • Re: (Score:3, Interesting)

      by ledow (319597)

      Surely it completely depends on the size of the company and the importance of the data?

      Losing a week's work is *NOTHING* for a small business holder if they have all the normal records in order (it might only take an hour to bring them back up to date), but having a daily backup might actually be a huge chore for them if it involves swapping disks, wearing out tapes, has to be manually initiated, the servers are only on during the working day, etc. Losing a day's work for a 2000-employee company may be a b

      • by bertok (226922) on Friday November 18, 2011 @08:28AM (#38096736)

        I hope you're a troll, because if you're not, this is just about the most insanely fucked up attitude to data protection I've ever seen.

        A blind "Let's just backup the whole server" isn't an effective backup strategy ...

        If I was your boss I'd fire you on the spot for being that monumentally stupid. The cost ratio of an additional ~20GB of tape compared the enormous cost to the business if the server can't be restored quickly and reliably is staggering. Think $1 saved while risking millions of dollars of potential lost work! (1.6TB LTO5 tape = $80)

        a) "what if you upgrade to a larger disk" -- there is no backup system on Earth that can't restore to bigger disks. Most systems can restore to smaller disks too.
        b) "what if the server is slower than others and people start moving their data to something" -- you back up both of them. Restoring too much is almost never as bad as restoring too little.
        c) "just blanket-backing-up is likely to lead to problems later on" -- no, it doesn't. Achieving 100% coverage ensures that no matter where you data was on a server (or which server), it's on a tape somewhere.
        d) "notifying IT of new things that need to be backed up" -- have you met humans? This never happens reliably, and can't ever be made reliable.

        Back in the real world, a 100% complete backup of a typical Windows server can be restored without knowing the password, to dissimilar hardware (even virtual machines), and without needing the "original install disks". When it's done, it'll boot up, maybe reboot once or twice to fix up its drivers, and then your server is back, working as it did before. Compare that to a "data only" or partial backup. Now suddenly you're chasing down design documentation, passwords, IP addresses, software, serial numbers, and you haven't even started to restore anything yet. The clock is ticking, and the customer is breathing down your neck.

        A week's work should take no more than two weeks at ABSOLUTE maximum to recreate

        Recreate from what? Memory? Including data that was 100% electronic, and never seen by a human? How do you recreate your emails? How do you recreate your audit logs? How do you type back in non-textual data like digital images or audio recordings? How would you even know what's missing?

        I'd rather have a decent monthly, than an imperfect daily

        That's a false dichotomy. The total data stored is the same, you're just altering the frequency. The same amount of storage is needed, the same bandwidth is needed, and it ends up costing the same.

        I'd rather have three backups a day than monthly backups. Losing a day of work could mean a contract fails to go through. I've been in a position twice now where users have come to me literally crying and begging to retrieve a document they only started working on that morning that they had deleted accidentally... at 8pm, minutes before a deadline for a multi million dollar deal. After experiences like those I've often set up incremental backup frequencies as rapid as 15 minutes.

        So lets recap... your sum total DR experience is you once walked into a poorly supported environment, and gave them some even worse advice, without ever being in a position to be responsible for an actual real world recovery.

        Well, take some advice from someone who's restored terabytes of data, and was responsible for the protection of over a petabyte spread across thousands of servers at over a dozen organisations:

        #1 There are no time machines -- you cannot go back in time to fix a mistake in your backup strategy after a disaster. It's too late. You've fucked up, it's your fault, and you can never, ever, fix it.

        #2 Back up everything -- I love genius IT folk who like to shave 1% off their backup times by excluding those useless temp and log files, also excluding 'useless junk' like their database transaction logs in the process.

        • by ledow (319597)

          I'm not sure if you've missed the point of the post entirely, which was stated in the first sentence, or got into a rant and went back to rantify it more afterwards. If you are a multi-million dollar establishment, I'm not suggesting you SHOULDN'T do dailies or anything else - because the cost of not outweighs the cost of. You're applying large enterprise IT to the world and that's exactly the point I make.

          The drive to go with the tape you specify can cost more than some COMPLETE small business IT systems

  • Easy (Score:5, Insightful)

    by MortenMW (968289) on Friday November 18, 2011 @04:30AM (#38095562)
    It's expensive, so management does not really want to pay for new tapes, a disk-based system or cloud backup. It requires personnel, which management does not want to pay for either. It's boring for the persons involved (who likes testing their backup?).
    • by Threni (635302)

      Also, it'll never happen to them. And it doesn't make them any money.

    • Re:Easy (Score:4, Insightful)

      by Billly Gates (198444) on Friday November 18, 2011 @05:17AM (#38095794) Journal

      Document the hell out of that!

      See if you can cause a server to be down. When the CEO screams tell him such and such wanted to save $400 and didn't think it was worth it save $40,000 an hour on lost productivity to save $400 now. Sorry that had to happen during that $ 1,000,000 deal you lost. Heads would roll faster thanyou can say fired. ... unfortunately, this makes a cloud all the more attractive that could end your own job :-(

      Companies are really stupid today! Normally I do not use exclamation marks, but the latest scaffle in Thailand where every single hard drive is made on earth in an all your eggs in one basket to save on 5% economies of scales cost, to companies producing junk, and now this drives me crazy. It bothers me because so many of us are out of work and these companies are penny wise but certainly dollar dumb. To me no tape backup in any fortune 2,000 company is the equilivent of having no fire insurance. A single disaster could cost you everything and it is so cheap when you rake in millions.

  • risk (Score:5, Insightful)

    by Tom (822) on Friday November 18, 2011 @04:45AM (#38095642) Homepage Journal

    Most companies have no risk management, and no clear picture of the risks their business faces.

    The result of "intuitive" risk-non-management is that the usual human flaws have full impact. Basically, aside from a narrow middle ground, all risks are wrongly estimated.

  • by Alain Williams (2972) <addw@phcomp.co.uk> on Friday November 18, 2011 @04:49AM (#38095662) Homepage

    The result is that people just expect them to work flawlessly -- not to fail. They also ignore other risks. I put in a machine at some customers a couple of years ago. They did not want to pay extra for backups -- ''yes, we will do that later''. They knew that I configured mirrored (RAID 1) disks. I set up a backup from one part of the disk to another and reminded them every couple of months that they would loose everything if it was destroyed or stolen.

    Then a few weeks ago another unit on the industrial was torched -- arson. I have finally pursuaded them ... I am putting in another machine on the far side of their factory that will take a daily rsync, and USB plugin disks that they will backup to weekly & take off site.

    These guys are not stupid in what they do professionally, they have an annual turnover of more that £1 million. Why does it take a fire at a neighbour to make them see sense ?

    • by identity0 (77976) on Friday November 18, 2011 @05:54AM (#38095982) Journal

      Then a few weeks ago another unit on the industrial was torched -- arson. I have finally pursuaded them ...

      Ahhh yes, the old "Nice data you have there, be a shame if something happened to it" trick.

      No need to explain, friend, we all know what happened there.

      I am curious if you did the deed yourself or contracted it out to a professional, though. ;)

      • by Culture20 (968837)

        I am curious if you did the deed yourself or contracted it out to a professional, though. ;)

        He probably suggested Sony laptop batteries. They took care of the rest themselves.

  • by bazald (886779) <bazald@zeniDEBIANpex.com minus distro> on Friday November 18, 2011 @04:49AM (#38095672) Homepage

    Seriously. Most people aren't willing to put in the effort to determine just how bad things would be if they had to resort to their (often inadequate) backups, and therefore they aren't willing to pay the time and capital to get adequate backups.

    If you want your company to get better backups, run a simulation of what would happen if something failed. What's the best recovery you could do? What business would you lose? Then calculate the probability of that failure occurring, and be generous.

    • by znerk (1162519)

      If you want your company to get better backups, run a simulation of what would happen if something failed.

      Quickest and easiest way to do this in a dramatic way?

      During a chat with the CEO, reach over and shut down a random server. Ask him where all the datas went, and how the company is going to operate without whatever data was on that server for however long it would take to recreate the data. Might get you fired, but might also make him reconsider the backup project he just axed because it would require him to put off that new car purchase for 6 weeks.

  • especially when combined with 'find' and 'xargs', in what is supposed to be a simple task.

    If you don't, you'll do something like what i just did ("worst typo in a decade"): you see, i was trying to update emacs and wanted to purge all the .elc files from ~/.emacs.d
    Unfortunately, through a bad typo, some miss-applied keyboard shortcuts, and rushing through without mounting a scratch monkey... what actually ran was effectively "find ~/.emacs.d | xargs rm".

    accidently deleted the 'grep'. Oops. 15+ years of elisp/etc destroyed.

    Was it backed up? Nope! Been meaning to check it all into git, but always put it off as a "minor, unimportant" task I'd get to later. Of course, we all think that way up until the disaster hits...

    *sigh*

    • by dutchwhizzman (817898) on Friday November 18, 2011 @05:26AM (#38095850)
      Mod me down for trolling all you want, he deserved the pun after not making backups for 15 years consecutively.
    • by Rockoon (1252108)
      Having no copies at all for such a duration is amazing.

      I may not have a proper backup of 100% of the source code I have ever written, but I simply cannot imagine ever not having copies of the stuff written before my last big system upgrade (unless a fire takes out everything.)

      It is simply unthinkable to not have a copy of at least 7 of those 15 years.. never in 15 years was a copy made.. seriously?

      This is beyond "backup" territory, because even people that dont "do" backups have burned their shit at
  • I think my record for getting things restored off tape is about 10%. It mostly seems to be a placebo.

    'Do we have backups of FOO?' 'Oh sure, we backup everything nightly.' 'Thanks, could you get me... FOO/BAR?' 'Sure. Justasec.'

    Two hours later I get the call. 'Uh, we're having some issues here, it'll be a bit longer...' Two hours after that 'How badly did you need FOO/BAR?'

    Either the automatic backup system was failing early on in the backup and had been for months and nobody had noticed the error condition,

    • Tape may have a failure rate, but if you test your backup systems like you should, and correct failures, build redundancy into the backup system and all that, you'd have at least three nines recovery rate, not 10%. Don't blame tape for this, horrible ratio, but blame yourself for not designing and testing an adequate system.
      • by bryan1945 (301828)

        To be fair, he wasn't doing the backups. Or is he supposed to take over the IT section while he is at it?

      • by Sarusa (104047)

        I'm not going to blame myself - it's not MY system, which works great; tape is the IT department's system. And that's the problem. The whole tape 'ecosystem' seems to be too fragile for most IT departments to handle. If you get great IT guys I'm sure they can run a fine tape backup system, but usually they know even less about computers than the other employees.

        Mirror to disk is much harder to screw up (though certainly possible!) and easier to verify. Both places I've been that had disk backups had no pro

      • by Lumpy (12016) on Friday November 18, 2011 @07:59AM (#38096544) Homepage

        I have had SDLT tapes I recovered from the floor of a server room after sitting for 3 days under water that read just fine in a drive after being dried out.

        If you get a decent tape backup it has built in redundancy, 48 tracks of data with CRC checking and redundant data written in case that CRC fails.

        Granted, I'd not rely on them surviving kicked around the floor, but a real tape backup system is far, far more robust than the under $1000 garbage sold at newegg.

  • Yeah, induction [wikipedia.org], folks.

    It's been 6 months since I began my startup. That's 3600x24x30.5x6 seconds, or 15,811,200. 15,811,200 seconds of no computer problems, and 0 with.

    Extrapolating out, that means infinite problem-free computin ??#@ NO CARRIER

  • Offsite backups are what a lot of companies don't do. They might back up to tape, but the tapes are stored in a pile next to the server. And they never test them.

    • Re:Offsite backups (Score:5, Insightful)

      by znerk (1162519) on Friday November 18, 2011 @06:15AM (#38096062)

      Offsite backups are what a lot of companies don't do. They might back up to tape, but the tapes are stored in a pile next to the server. And they never test them.

      Agreed. One of the many phrases in my litany on customers backing up is "Yes, that safe is fireproof. For paper. Plastic melts at a much lower temperature than paper burns." Not only do not enough companies run backups on a regular and timely basis, but too many of the ones that do run backups don't see the need for having the storage media offsite.

  • by pe1chl (90186) on Friday November 18, 2011 @05:29AM (#38095866)

    I'm not sure how everyone gets so ecstatic about those cloud backups. When we would need to send all our data over the internet connection it would take an unworkable amount of time to complete the backup.
    Even to the local LTO-4 drive, which runs at over a gigabit per second, the backup takes an appreciable amount of time.
    Cloud backup may be good for a 3-man company doing document editing, but with the amounts of data that are common these days, and the speeds of internet connection that you normally have, I don't see it as a realistic possibility.

  • Slashdot (Score:5, Interesting)

    by identity0 (77976) on Friday November 18, 2011 @05:45AM (#38095946) Journal

    I've always wondered if /. bothers to back up the stories and posts...

    Anyone here know?

    How about for other 'trivial' sites, like reddit, youtube, etc?

  • You need to think about what data you are protecting, why, how you RESTORE it, and whether it needs an offsite solution at all.

    If you do that properly you knock a zero or two off your technical volumes, every time.

    Conversely, it's seldom worth doing that at $SMALLCO because the entire enterprise fits into a Terabyte or less and storage is cheap. The danger is that throwing it onto something offsite - whether it's disk, tape, cloud, or ferromagnetic core memory - can lead to lazy thinking about what happens

  • ... because hard drive costs have come down so much it's just cheaper to buy a bunch of hard drives and mirror like crazy. Another factor is speed. Backup up costs time and time is money. So it makes sense that more and more organizations have moved to to mirroring/RAID solutions.

    RAID is pretty damn robust these days also and with drives as cheap as they are you can create many mirrors at once. I've owned raid 5 over the past 7 years and I've never lost data drives have always died 1 at a time on a disk

    • by pe1chl (90186)

      Wait until your boss deletes that important document and your RAID system has deleted it on all drives in the array immediately at his request.

      Or your business application is slowly corrupting the database and it is noticed (or finally confirmed) only after 3 weeks of use.

      At that time you want to be able to get old data back. This is not something your array is going to be able to provide you.

      • by Lumpy (12016)

        "Wait until your boss deletes that important document and your RAID system has deleted it on all drives in the array immediately at his request."

        Do you know how RAID works? it stripes the data across ALL drives. if you think RAID5 keeps separate copies you are a little nutty.

        RAID 1 is mirroring, but nobody at all uses it other than home users that are paranoid and like paying 2X for their storage space.

        and to stop idiot bosses, you put in place a RCS. revision control does wonder to solve the "oops I de

    • by javanree (962432)

      Sure, if you only have like 1-10TB harddisks are fine...
      But how to do you handle backups of say 50TB of data every week? Awful lot of disks to swap, copy times will be so long you'll need to start the next weekly while the last job is still running etc etc.

      Tape still has it's place. It's not the universal solution for all backup problems, but for large datasets it's still king.

    • by Gordonjcp (186804)

      RAID won't save you from the ohnosecond when you realise that you actually typed "rm -rf / tmp/*" though.

    • by znerk (1162519)

      RAID is no substitute for backups. Yes, RAID5 will handle a single disk crashing. ... but if the building burns down, "where did my datas go?"
      if more than one drive fails simultaneously...
      if a drive dies while you're still resyncing your array from the first disk crash...
      if you don't notice you have a failed drive because you didn't install the disk-monitoring tools...

      You can mirror all you want to, and if the data never leaves the room, it's not backed up.

      I guess a potential solution using only RAID would

      • by Lumpy (12016)

        "RAID is no substitute for backups. Yes, RAID5 will handle a single disk crashing."

        You know how many times I have seen a raid 5 go down hard with a single drive failure? too many to count.

        Raid 5 lost a drive, no problem the hot spare will swap in and the rebuild starts..... and then drive #2 fails..... all the data is now utterly hosed as it failed during a rebuild, or better yet, the freaking hot spare was an old dead drive that someone failed to pull out of the cage during the 2 years when they cruise

  • Complacency (Score:5, Insightful)

    by Tastecicles (1153671) on Friday November 18, 2011 @06:12AM (#38096054)

    To most PHBs a computer is a toaster. That's right, an appliance. Nothing more. Most have no idea of the nursemaiding a computer needs, or how vulnerable company data is when it's reduced to 0s and 1s. Until the "unthinkable" happens and the toaster breaks.

    Now. I used to be in the biz of disaster recovery. It was lucrative when consumers who valued their potentially lost data found me and asked me "Can you do it?" and I said "Let's have a look". My success rate was something like 40%. Surprisingly (for me anyway), my better corner was recovering data from physically damaged flash drives. Out of several dozen of those I only had one that I couldn't recover anything from. Hard disks are a different animal, and the whole thing can be frustrating when you sit back for a minute or five and consider how much easier your work would be (and how much hardware you can sell) if your client had the benefit of hindsight and someone around who knew the shit of which he spoke - so a week-long recovery project that may or more than likely may not work turns into a two hour exercise in restoring on new hardware and sending the happy client on his way. Now, that is an exercise in getting repeat business through recommendation.

    As far as "accidental" deletion of data: there is only so much you can do to protect the stupid from themselves - on a network share, for instance, you can deny users the right to delete files. Job done. In an environment that is supposed to be secure, that's a good start. VSC is another handy tool but you don't need to tell the stupid that their fuckups are (sort of) covered - it breeds complacency and does little to nothing to train responsibility.

  • by Lumpy (12016) on Friday November 18, 2011 @07:27AM (#38096380) Homepage

    I backup to backup server every hour and then spool to tape every night.

    Once a week that weeks tapes get sent offsite and old 3 month old tapes are returned for re-use.

    If the company's IT department is NOT doing daily backups then it's IT department is ran by an idiot.

    If management will not pay for it, then they need to be told, "so everything you do today does not matter and has no value? because that is what you are saying when you say backups are not important. Hard drives are not reliable, if they crash tonight and we lose everything, how much money do we lose?"

    You have to talk to them in management speak... AKA money.

  • by MichaelCrawford (610140) on Friday November 18, 2011 @08:41AM (#38096830) Homepage Journal

    I expect a lot of companies are just like my mom. having never experienced data loss, she doesn't see the point of backup.

  • by hesaigo999ca (786966) on Friday November 18, 2011 @10:10AM (#38097576) Homepage Journal

    >showing 10 percent were only backing up to tape once per year
    You mean showing 10 percent were only backing up to tape once per year to a paying third party vendor....
    instead of in house which is cheaper when you know what you are doing...

    I love when they twist these headlines to make them more attractive..

  • Pretend a guy prevents the 9/11 attacks by requiring all airliners to have a bulletproof, locked cockpit door. Hundreds of millions of dollars are spent, and prices go up. Everyone complains about government regulation. The attacks never happen. The instinct was that we just wasted a bunch of money. We spent hundreds of millions on H1N1 vaccine for an outbreak that never happened. But if it did, the vaccine would have saved many lives and prevented great financial loss. But it never happens, so it's seen as lost money.

    You push for backups, and you spend tens of thousands on it for years. There is a disaster eventually that you recover from quickly, but perhaps you aren't around to take credit for it. Or you restore and everyone thinks that it was supposed to because we spent so much on it. No one will think, "Well, we just saved millions of dollars of downtime because of the backup." The thinking is just, "Oh, well it was supposed to. We spent so much on it."

    This kind of myopia is commonplace in the world. We can invest money to treat everyone for chronic illnesses such as diabetes, high blood pressure, and high cholesterol with really cheap drugs. We can vaccinate everyone from childhood diseases. "Oh, but vaccines cause autism." Yes, but your child isn't paralyzed from polio. The avoided disaster is never quantified; only the cost spent shows up on your calculations.

All warranty and guarantee clauses become null and void upon payment of invoice.

Working...