Forgot your password?
typodupeerror
Data Storage

Ask Slashdot: Best Offline Storage Method For Large Archives? 397

Posted by Soulskill
from the stone-tablets-and-a-sharp-chisel dept.
An anonymous reader writes "I have a collection of large projects (Indesign files with associated images), which are typically 40GB to 60GB each. In this current climate, what is the 'best' method of archiving these? Spinny magnets? Solid state drives? USB? Tape? Blu-ray? All have pros and cons and price considerations. If I remove the price issue (my data is important to me), does this change the choice?"
This discussion has been archived. No new comments can be posted.

Ask Slashdot: Best Offline Storage Method For Large Archives?

Comments Filter:
  • Rotational media (Score:5, Interesting)

    by BWJones (18351) * on Wednesday July 20, 2011 @12:13AM (#36819402) Homepage Journal

    For this project [utah.edu], we have multiple multi-terabyte (5-18 terabyte) datasets that need backup. We have online and offline strategies and the offline strategy is simply multiple, redundant copies on hard drives stored in static proof containers onsite and off site.

    Hard drives are *very* cheap all things considered, are easy to store, take up very little physical space and if things go badly, restoring from them is faster than just about any other method. For datasets in the GB range, its a no-brainer to go with hard disks.

    • by Mad Merlin (837387) on Wednesday July 20, 2011 @12:32AM (#36819522) Homepage

      I concur on this point, online storage really makes the most sense. Cheap, high performance (for sequential read/write) and easily expandable. You can get a single machine with dozens of SATA drives in it (including the drives) for way under 5 figures. When drives fail, they're simple to replace, and every couple years, migrate the whole thing to newer (faster, bigger) drives. Mirror your data unless you don't care about it. RAID 1/10 for really small datasets (2-4 drives), RAID 6 for moderate size datasets (5-10 drives) and RAID 60 for anything bigger.

      A very important note to keep in mind... stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card (buy it up front, they won't exist in a couple years). The same goes for fake RAID (ie, software RAID driven by the BIOS), but s/controller card/motherboard/g;. Pure software RAID (ie, using mdadm) is a safe bet.

      • A very important note to keep in mind... stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card (buy it up front, they won't exist in a couple years). .

        I have to disagree... Adaptec raid card -they stand the test of time. Hell i can still buy a 2940 controller card if I want one...new!

        • by Spazmania (174582)

          I've used a dozen different brands and models of hardware raid cards. The only ones that haven't lost my data are the defunct Mylex DAC960 and HP's Smartarray controllers.

          Had a problem with the mail server recently where a malfunctioning drive kept killing the scsi bus, causing the Smartarray controller to see every drive except that one as faulty. Once I figured out the bad drive, recovered without losing the data. Had similar problems using Adaptec and LSI cards a few years ago. Data gone. Maybe the curre

          • by drsmithy (35869)

            Linux software raid is also superb for reliability if you can tolerate the performance of software raid.

            In most cases. with modern hardware, software RAID has superior performance to hardware RAID.

            • by Spazmania (174582)

              Not really.

              If you sprung for hardware raid without the battery/capacitor needed for the write back cache then you're a fool.

              With the write back cache, the instant return for data committed to the cache makes the hardware raid insanely faster than software raid for most disk loads.

              With a write-through cache (no battery) a software raid 1 is usually faster. You have to have a pretty crappy card before a raid 5 or raid 6 is faster on the main CPU than on the dedicated card. The extra computing on the main CPU

              • by drinkypoo (153816)

                The extra computing on the main CPU and the extra I/O overhead to multiple devices instead of one takes its toll.

                The first part yes, the second part no. The same amount of data is going across the bus either way.

          • And for the record, an Adaptec 2940 is not a raid controller. It's a plain jane SCSI card -- something which unlike RAID controllers Adaptec did very well.

            My point was that they still make them. Not all hardware is gone in a few years.

        • by Z00L00K (682162)

          The downside with Adaptec is that they don't have the best performance - even with raw access to the disks through the controller.

        • by the_B0fh (208483)

          *WHAT* century are you living in? In the 1990s, Adaptec was a good brand. Not any more. Not just not good, but actively sucky.

          http://marc.info/?l=openbsd-misc&m=125783114503531&w=2 [marc.info]
          http://marc.info/?l=openbsd-misc&m=126775051500581&w=2 [marc.info]
          http://marc.info/?l=openbsd-misc&m=128779369427908&w=2 [marc.info]

          (Read 'em in order)

          Adaptec: another way of saying "my data is not important"
          Adaptec: unsafe on any platform.

      • by lucm (889690)

        > stay away from hardware RAID! When your controller dies, so does all your data, unless you have an identical spare controller card

        I totally agree. On some systems even a broken RAID-1 cannot be recovered using a different controller.

        It's a good thing that more and more filesystems are going beyond RAID (such as zfs or btrfs) and even storage unit are moving to block replication.

      • I have had two disks in a RAID 10 fail me directly after each other, once. Guess which two? Yay!

        Especially for backups where write speed is not much of an issue, you want RAID 6 or above. Never RAID 10.

      • I cant believe my ears. Someone is asking for advice on archival media, have suggested that tape is an option and price is not a big factor, and the response is "yea, you should buy less reliable, more expensive rotational media".

        Why on EARTH wouldnt he pick up a cheap LTO3 drive (which can be had for $200 these days), grab a couple $20 tapes (holding ~300GB), and call it a day? Theres your multiple copies, LTO supports WORM tech, and youre basically guarenteed that you will be able to read the tapes for

    • Even on a small scale this makes sense. The easiest is 2.5" external drives ($100 for 1TB). This avoids the mess of power adapters. If you need significantly more storage, you may want to consider a dock ($50) and internal desktop drives ($80 for 2TB). Consider this: you can buy from anywhere a USB adapter that will plug into a 20+ year old drive and any OS will mount it. Wish I could say the same about all my removable media...

      Traditionally the way to do this is with tape. As you replace the drive (a

      • by adolf (21054)

        Media doesn't age as fast as you think.

        I've got a USB 3.5" floppy drive that I bought about six years ago. I've only used it a few times, but it's a glorious little thing when it's needed.

        I've also got a couple of 5.25" drives (both low and high density, which is important due to the difference in head sizes), but those are available (and cheap!) too.

        I've got a box full of QIC-80 and DDS tapes, both of which (AFAICT) are abandoned standards. I don't have a drive for either of them anymore, but that's not

      • by evilviper (135110)

        you can buy from anywhere a USB adapter that will plug into a 20+ year old drive and any OS will mount it. Wish I could say the same about all my removable media...

        Maybe if you used FAT. Any other file-system at all, and you're struggling.

        Ext2 is fine on Linux, but I haven't found any other OS with a halfway decent implementation, though. Sure, there are plenty of half-working implementations, but fsck on ext2 outside of linux? Forget it. UFS/FFS might have worked, but interoperability between implement

    • by lucm (889690)

      I agree with that, unless the budget is insanely low hard disks are a no-brainer. They are also easier to encrypt than tapes and offer a much higher rate of reuse.

      • by mlts (1038732) *

        Just make sure your encryption program is available, perhaps as freeware, or use a utility that is widespread and easily gotten. I've seen fancy encryption programs for drives that would be useless if there ever was a restore needed just because the license keys likely wouldn't work.

        For encryption, I'd go with LUKS, or TrueCrypt on a disk level. For file encryption, gpg is solid, perhaps tar, bzip2, and gpg, although you might need a utility for error detection/correction to repair any data lost due to ba

        • by profplump (309017)

          If you put tar archives on tape in 1976 they'd still read with modern GNU tar on a standard linux distro today, assuming you had a working tape drive. There is *no* random-access filesystem for which the same thing can be said. Certainly not all tape formats survived the test of time, but if you choose wisely tape a safer bet for longevity than any random-access file system, if for no other reason than the format is physically easier to read and the filesystem is easier re-implement.

          That being said, for dat

    • Ok, your projects are about 50GB each, so you can fit about 20 of them per Terabyte. How many of them do you want to keep? If they're something that you generate 100 of them per day, all year, you're looking at a much different solution than if a project takes you a month to tweak all the pixels lovingly by hand.

      For a few Terabytes, just use 3.5" hard disk, make backups, keep one copy offsite. If you want to keep 10 or so TB handy and online, maybe you'll want to do a RAID thing, or maybe you just want J

    • by vegiVamp (518171) on Wednesday July 20, 2011 @08:04AM (#36821736) Homepage

      Fully concur, and let me sum it up: The best type of long-term storage is "redundant".

  • by sirsnork (530512) on Wednesday July 20, 2011 @12:14AM (#36819408)

    You probably need to define "best". How long do you really want to keep them for, and in what sort of environment.

    Traditionally the answer is tape, and probably will be in your case too for files of that size. Optical isn't proven enough (at least for the sizes your're talking about) to be trusted, and HDD's need to be run up fairly regularly to keep working.

    • Why not punch cards? Huh huh... so clever!
      • by Z00L00K (682162)

        You can probably print it on archive-persistent paper instead in binary form with ECC coding. Then scan it back into the system whenever necessary.

        The disadvantage would be the huge amount of paper and storage facilities needed.

        Personally I would go for a solution involving 2-3 independent nodes with a RAID array on each and then replication between them. At least one of these nodes shall be on a remote location to cover for any cases where a fire or theft occurs.

        If you don't feel secure enough then you sha

    • by afidel (530433)
      For that small of files the most reliable inexpensive method is probably a small LTO2 library with each backup going to multiple tapes. $20 per tape with each capable of holding quite a few projects (200GB native). I just checked ebay and there are a ton of 2x LTO2 libraries for ~$500.
    • Stage it to a HD for ease of access, back up the HD to tape for long term storage.

      Dont forget to test your backups! Just because your software "confirms" that it wrote the data does not mean you can restore the data.

    • by Pooua (265915)

      Yeah, tape is the traditional answer, and still gets a lot of support in the industry. Unfortunately, I have never seen a successful restore from tape. I understand that my experience is not unique. Many companies have made regular tape backups for years, but never tried to do a restore, until that day came when they needed it. Then, they found out that it didn't work.

      I've been backing up data to hard drives for 20 years. I've rarely had a problem getting my data off a hard drive, and redundancy has been a

      • by Joce640k (829181)

        I have never seen a successful restore from tape. I understand that my experience is not unique. Many companies have made regular tape backups for years, but never tried to do a restore, until that day came when they needed it. Then, they found out that it didn't work.

        I've been backing up data to hard drives for 20 years. I've rarely had a problem getting my data off a hard drive, and redundancy has been a big help. I maintain about 4.5 TB of data on about 7 TB of external HDDs.

        I'm 100% with you on all that. Tape looks/sounds good on paper but never seems to work when you need it most.

        Hard disks have never failed me so far.

        PS: Always test your backup system to see if it can restore.

      • I got lucky and managed to recover data off of a DAT tape I made 10 years ago. The key is that you have to have the backup software that created the archive to begin with! Also, tape was NEVER intended to be a long term backup medium. Moisture collection over time is tape's biggest enemy and slowly destroys the binder and lubricant in the tape. Tape also requires contact with a read/write head which adds wear and tear, plus the drives have a lot of moving parts (high rate of failure).
    • by Xacid (560407)

      I'm actually a little surprised at all the comments voting for tape instead of huge disk arrays or other fancy such configs, but it makes sense.

      I'm curious - have any of you dealt with these guys? http://cache-a.com/index.php [cache-a.com] It looks like they're geared for video work, can be networked, and runs on tape. Think it could be handy for what the OP is after (despite being graphics as opposed to video)?

  • by mlts (1038732) * on Wednesday July 20, 2011 @12:15AM (#36819414)

    BD-R disks are an idea, and relatively inexpensive, but your best bang per buck would be large removable disks in the 2-3 TB range. The reason I state "disks" plural is for obvious reasons.

    I would also use a program like WinRAR with a recovery record, or one of the PAR utilities used for USENET to store your files in. This way, you can tell if there was file corruption, and have a good chance of recovering from it.

    For serious stuff where money is less of an issue, I'd consider a LTO-5 tape drive and multiple tapes. Tapes tend to last longer than HDDs because they have very few moving parts.

    Don't forget to see about copying your archives to new media every couple years. It isn't uncommon to be able to pop a 10+ year old tape or HDD in and pull off the contents... but it isn't uncommon either to find the HDD clicking, or the tape full of hard errors.

    • by AmiMoJo (196126)

      I think the bottom line is that no medium is bulletproof. If you really care about the data and money is no object then a combination of at least two different mediums is the way to go.

      Aside from the usual suspects like tape and HDD I'd suggest looking at flash memory. Expensive per GB but also not prone to mechanical problems. Most flash memory states data retention for 10 years, but it is a little bit more complicated than that. Every time you write data to a flash memory device it "refreshes" and the 10

    • If you want true longevity of archives, it isn't about finding a format that will not ever die, because they all can. It is about making copies. The brilliance of digital storage is perfect copies for an unlimited number of generations. So you take advantage of that. Have more than one backup, and test the backups. If one fails, make a new copy from the good data. Also, check the expected life of the backup medium, and replace it with new copies when it starts to age.

      Along those lines, to keep it useful, ma

  • by bynick (1038382) on Wednesday July 20, 2011 @12:25AM (#36819472)
    Screw tape... you pay $2,000 USD for the drive, $50+ per tape for a couple of hundred gigs. Go with bare drive external: Install a trayless SATA bay for 3.5" hard drives... this will run you $12. Buy some bare SATA drives.. these run $50 for 1TB and are available up to 3TB. I buy bare drive hard cases for about $3 each. My Intel ICH10R on-board RAID controller supports hot-swap -- so in effect it's a big 3.5" floppy.. that's right. If your tape drive breaks, you're out another two grand. This is far less expensive, faster, higher density, and random access. In addition, you can boot from it. Want RAID0? Install two trayless SATA bays for a total of $24 and back up in pairs.
    • by lucm (889690)

      > Screw tape... you pay $2,000 USD for the drive, $50+ per tape for a couple of hundred gigs

      What brand of hardware are your buying? Rolex? A LTO-5 cartridge is about 60$ and unless you format it with a very, very big cluster size it will store 1.5TB, not 200GB.

      > I buy bare drive hard cases for about $3 each

      Ok, I see. For tapes you shop at Rolex but for your "neat setup" you browse eBay until you find one of those Hong-Kong power sellers that ship on the Pacific Princess Express Line. Makes sense.

  • We really need..... (Score:5, Informative)

    by pcjunky (517872) <walterp@cyberstreet.com> on Wednesday July 20, 2011 @12:28AM (#36819492) Homepage

    Eternally Yours, The case for the development of a reliable repository for the preservation of personal digital objects.

    http://explorer.cyberstreet.com/CET4970H-Peterson-Thesis.pdf [cyberstreet.com]

  • by fuzzyfuzzyfungus (1223518) on Wednesday July 20, 2011 @12:28AM (#36819494) Journal
    Depends on price: HDDs are crazy cheap, for the capacity; but untrustworthy. However, thanks to the cheapness, redundancy, preferably in multiple locations, periodic testing/copying to newer disks/etc. is fairly affordable. Make sure that you have(either manually, at the utility level, or at the FS level, hashes/checksums) and hope for the best. LTOs are rather more durable, having fewer moving parts in the storage media; but the cost of entry is substantially higher. All the same principles apply, though.

    There are no truly reliable storage mechanisms for large quantities of digital data, only storage mechanisms cheap enough that you can duplicate your way to reliability.
    • by hedwards (940851)

      One possibility would be keeping an offline copy on something along the lines of a DROBO and then storing a copy with Crash plan. It's expensive to send disks, but for images that size it's just about the only way to ensure that the data is being properly backed up off site.

      For 125 the OP could back up up to 1tb of data by mail. Doing that weekly is probably a good deal and between those chosen days a copy could be taken home.

      Ultimately with data in that kind of volume there is no good way of dealing with i

  • by triffid_98 (899609) on Wednesday July 20, 2011 @12:34AM (#36819538)
    You can't argue with Tape. It's been proven to last since the 1960's if kept in a climate controlled space (dry/cool). Just make sure to keep a spare tape drive handy (just ask NASA), because spare parts for 40 year old tape drives are surprisingly difficult to locate.

    Optical isn't even close, assuming you're talking burned discs. Taiyo Yuden claims a 70 year shelf life, but they have only been around for what, 8 years tops?

    Hard drives are an option if you've built a redundant array, but even with that you're still going to be out of luck if you burn up your raid controller.
    • by hitmark (640295)

      Just wish that tape drives where a bit more available in prices ranges outside of the business stuff. Then again, i guess that is where optical come in. Burn two copies, and store them cool and dry. In the end, any media that allow the separation of rw hardware and the actual storage object without requiring a clean room is more reliable imo. This because rw hardware failure can then be recovered from.

    • by timeOday (582209)
      I would just copy each project to two separate external 3TB drives and store one offsite. Each drive is then large enough to hold about 50 projects of the specified size. This gives you redundancy, protection against accidental modification/deletion (unlike RAID), and no single point of failure, and no reliance on obscure or vendor-specific technology. Oh, and it's cheap. So what's wrong with it? Too easy and inexpensive?
    • by Lando (9348)

      I don't do enough tape to know what the market is. I was burned so badly on colorado tape drives and qebit? that I've never seriously considered it again.

    • Funny, everyone says the 15 year old CD-Rs I have shouldn't work anymore. They all read and verify just fine despite being first generation dyes. DVD-Rs are better then CD-Rs simply because of the added top lacquer coat they all come with. The dyes are also mature and stable and drives universally available, something hasn't been proven with BD-R yet.
  • Last I looked into this, the best format in terms of reliability was magneto-optical. It heats up the disc with a laser before the magnetic bits are able to be manipulated, so it's unlikely to be corrupted by only magnetic interference or only light/heat. You can get a 9GB rewritable disc or 30GB write-once for ~$50.

    There's also tape, which has massive capacities, but every anecdote about tape I've heard ends in "but the tape backup was partially/completely corrupted". Make two tape backups from the source

    • by afidel (530433)
      Dude, I've put 6400+ LTO 1-4 tapes through 4 different libraries over the last 5 years and had exactly 3 failures to read, two of those cartridges were dropped and so the failure was not unexpected, the third had a manufacturing flaw where an extra piece of tape somehow got into the cartridge. We verify every tape after writing using a different drive in the library and do restores almost daily so it's not like I'm blindly trusting the backup reports. I've restored data from old DLT IV carts that were almos
      • by dbIII (701233)
        I'll second that, only with nine track reels from the 1980s that really should have been stored a bit better (hot, humid storage shed for 30 years) but have still been recoverable with care. Out of around four hundred there were two tapes with problems. What should have have been done with those tapes over the years is a copy every few years onto newer media - but in the case of all those tapes they are theoretically a third copy made for transport and the data belongs to the client. It's a bit too much
  • If you have more than a few of these projects, SSDs are not yet a good choice for backups. You say price is no issue, but I doubt you'd want to buy 50 SSDs at their current prices. I'd suggest a few "spinny magnets," perhaps in an array if you need more apce than a couple of terabytes. Pros: low cost per terabyte, reasonable transfer speed, decent reliability, easy to implement as a tried and true technology. And of course for added safety, mirror the drive/drives to a second set. USB (if you are refering t
  • by stox (131684) on Wednesday July 20, 2011 @12:51AM (#36819652) Homepage

    How long?

    What is good for a decade may not be good for a century, and vice versa.

    For millenium+ archives, nothing beats punch cards.

    • by artor3 (1344997)

      For millenium+ archives, you better just make it a book. Computers have been around for how many centuries? What's that you say, approximately zero? Well, then we probably shouldn't assume that they'll look the same in ten. Heck, even your writing will probably look like "Hwaet! We Gardena in geardagum, theodcyninga, thrym gefrunon, hu dha aethelingas ellen fremedon," to people in 3011, but there will probably be historians who can manage it. Maybe there'll be historians who specialize in all the diffe

  • Torvalds quote (Score:4, Interesting)

    by bmo (77928) on Wednesday July 20, 2011 @12:59AM (#36819690)

    "Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it" - Linus Torvalds

    --
    BMO

  • Dedup or Tape (Score:4, Interesting)

    by lucm (889690) on Wednesday July 20, 2011 @01:00AM (#36819696)

    If price is not an issue, a great solution is to go with a data-deduplication device (such as EMC DataDomain or IBM Protectier). If you were to host one unit in your basement and the other in coloc environment far from your home, you could setup replication and have a very reliable archive. Coloc of a 1U device can be quite cheap, I have one of them for which I pay less than 100$ a month.

    If you have a smaller budget, then the best cost-benefit is still found on tape, and it can even work in case of network disruption. Like Andrew Tannenbaum said: "Never underestimate the bandwidth of a station wagon full of tapes". A single LTO-5 tape is very cheap (50-60$) and can store 1.5TB (can easily double that with dedup).

    There are other interesting technologies out there, such as MAID, which you can use as a VTL with a good backup software to maintain a reliable archive, however cheap disks are cheap and in a MAID configuration they might not last as long as typical disks because of the on/off behavior.

  • I guess this isn't a very popular suggestion. And you seemed to imply you wanted a local archive for your data, something you do yourself.

    I would just a large iSCSI NAS. 2TB Drives are really cheap these days. FreeNAS even lets you flag a drive as a hotspare so you don't have to as much about failures.

    Then back this NAS up to at least two online storage services. Make sure they're not both the same thing on the back end (like amazon's S3). Actually Carbonite personal can't distinguish iSCSI from a local dri

  • I know it is a PITA, but duplicating across pairs of hard drives is cheap per GB, and allows you to move the data to an offsite location / *SNL Al Gore Voice* lockbox. I have ~2TB of video project data that I store using this method.
  • OP: "If I remove the price issue (my data is important to me), does this change the choice?"

    ME: If price isn't an issue then you don't choose one, you choose them all.

  • Lots has been written on the subject of archiving, and with lots of valuable and eventually irreplaceable data, it would pay BIG dividends to read a few books and look at some of the companies that manage data for their living.

    Others here have noted the variables with respect to media, hardware & software and the fact that over time they all change and eventually become obsolete. Then comes the factors of where you store it and how many places do you store duplicates in to prevent fire, flood or whatev

  • Hardware gets cheaper and better by the minute. That means your choices are degrading your chances to recover archived information as soon as it enters storage. It's a paradox. I have no answer to that; I've been bitten by storage on Apple ][+ floppies which can't been read on any current hardware I've got. Hopefully, there's an aftermarket for people who can afford to read the obsolete hardware of the past and transfer it to the nonexistent hardware of the future. Maybe there's a standard that won't

  • by KahabutDieDrake (1515139) on Wednesday July 20, 2011 @04:13AM (#36820660)
    I'm seeing a lot of really goofy suggestions in here. I'm going to make my own. First, let me say that my last job was to create massive image archives sourced from disparate media, and store them, permanently. Massive, as in 30tb a year. (maybe not that massive, but we were a tiny company, with a matching budget).

    First, let me tell you what won't work. Optical media. Just DON'T. It's unreliable, slow and generally a pain in the ass. I worked at a place that burned 150 CDs a day for distribution, we had consistent failure rates within 20 days of 50%. Granted, that's using the cheapest possible media, but that's still awful. Further our "archive" had thousands of discs in it, was stored well, and as a whole, had a 41% failure rate over 10 years. Optical media is crap for long term storage.

    Something else that won't work, TAPE. I know, heresy. But listen for a minute... do you know anything about tape? Ever used it? No? Then don't touch it, unless you plan to hire someone that is an expert to build out the system and keep it running. Were you planning to hire a full time systems manager? I didn't think so. Alternately, if you happen to have experience with tape, hell, use it. You can't beat the density or reliability.

    Now, a suggestion that does work. Build your own NAS (or buy one if you don't have the chops to build it). You ought to be able to build/buy a 5tb array for under 3k, give or take. It will quietly hum along in the closet doing it's thing for pretty much the next few years. After 3 years, start a swap program to replace each and every hard drive. Doing this all at once allows you to store the old raid in cold storage (box it up and stick it in the corner). Doing this at the rate of one drive per month allows you to absorb the costs a little easier. Continue forever.

    Now, if you are really nuts, and you actually think your data is valuable (you know, like you can trade it for money at some point), then you build out the NAS, order three of them, and keep one at your mom's house (or wherever), then you buy co-lo rack space and put the third unit (did I mention you need 3?) in there and sync all three as often as you can afford the bandwidth. This is, for all intents and purposes, how google backs up data. 3 systems, in 3 locations, each with a complete copy of the data. It's not exactly CHEAP, but neither is redoing all that work.

    I'm going to leave out suggestions like using a kodak image writer to burn the images to microfilm that is digitally indexed. Why, because you don't know the first thing about a system like that, and because you want "backups" not permanent archives. Also, you can't afford this method. I'll also skip the really wacky shit, like using BD discs, or SSD arrays (in the terrabyte range? Fuck off$$$), or anything that involves the clouds.

    Storing relatively large groups of data has been dirt cheap and easy for the last 5 or so years. Even before that it wasn't that hard. Don't invent a difficult system, or buy into enterprise gear. You don't need difficult, and you don't need a NAS that performs 100,000 IO ops a second with a fiber channel back haul. You need a couple of raided drives in a box in the corner, powered up pretty much all the time.

    Oh yeah, and do you know the single greatest cause of HDD failure? Cold storage. TURN THE FUCKING THINGS ON, and leave them that way. They last MUCH, much longer. God it was hard to teach people that concept at my last company. No, putting the drives in a box in the storage locker does not make them last longer, in fact, they started failing the minute you unplugged them. (yes, I know, physical shock is probably actually higher up on the list, as is manufacturing defects, a little hyperbole never hurt anyone)
  • by jcoy42 (412359) on Wednesday July 20, 2011 @04:17AM (#36820684) Homepage Journal

    There are only 2 real solutions if you want real long term storage. The first is you become Linus and just dump it on a server and let the rest of the world back it up, and the second is you make your data a religious text somehow. Because those guys with translate it for centuries to come, even if it means sitting 50 dudes in a room for 3 years with nothing but a feather, ink, and parchment.

    come to think of it, same thing.

  • Sorry. It's not going to be popular with the Slashdot crowd, but dumping it onto a cloud storage service seems to make the most sense.

    I keep seeing ads for a service that gives you 2TB backups for £5/mo. For that you get full redundancy, and let them worry about replacing broken hardware etc. Cheaper than buying the hardware yourself, over the first year, and bound to get cheaper.

    If you're genuinely worried about your cloud provider having a catastrophe of some kind that their own fault-tolerance appr

  • Single-layer BD-R disks and 2TB SATA disks are currently matched at $0.04/GB. I will assume that the OP's data, which contains images, is already compressed sufficiently.

    The BD-R disks have an unknown livespan and the OP's dataset would have to span 2-3 disks per project. The 2TB disks would hold multiple projects. There is an argument to be had that it is less expensive and more reliable to use the BD-R disks from the perspective of adding a single parity disk. The loss of any disk set would lose that

  • by BigBadBus (653823)
    You could adopt the British Broadcasting Corporations approach to valuable archival holdings c.1972: thrown everything into a furnace.
  • If it's just you, and just one computer, why not carbonite (or another reputable online storage service) AND ALSO 1 or two external usb hard drives, keeping one off site and periodically rotating them. Like, weekly.

    If carbonite implodes, you have the hard drives. If you lose one hard drive, you have the other. If you lose both hard drives you have carbonite.

    If you never lose the working data, then you aren't out TOO much, as carbonite is not TOO expensive and external usb hard drives are also reasonable
  • The OP failed to mention how many of these things needed archiving. A couple hundred? Redundant disks (don't even bother with RAID) spun up once a year or so. Ten thousand? Tape. No question. It has proven and well-known long-term reliability. But you must meet the media's storage requirements to achieve the media life specs. (If you can't do that, there are any number of off-site tape storage places that can.)

  • My home media collection is around 20TB. For the longest time, I was dealing with redundancy by just maintaining a second, synchronized set of file servers. Each server has either a 16 port SAS controller or two 8 port controllers and a total of 16 storage drives in RAID6 including the hot spares. Each machine probably cost me $2500 to set up and I have four of them. And that's with getting the RAID cards and rack chassis from Ebay.

    The truth is, the chance of having a non-recoverable error while doing a RAI

"Say yur prayers, yuh flea-pickin' varmint!" -- Yosemite Sam

Working...