Best Way To Store Digital Video For 20 Years? 805
An anonymous reader writes "My kid is now 1 year old and I already have 100G of digital video (stored on DVDs, DVD quality) and photos. How should I store it so that it's still readable 10 to 20 years from now? Will DVDs stil be around, and readable, 10 years from now? Should I plan for technology changes every 5 to 10 years (DVD->Blue-ray->whatever)? Is optical storage better, or should I try to use hard drives (making technology changes automatic)? And, if the answer is optical, how do you store optical disks so that they last?"
Re:CDs are still readable (Score:5, Informative)
no they're not. ever hear of cd rot?
store everything on hard drives, with duplicate backups stored off site.
Gold Disks (Score:5, Informative)
Claim up to 300 years.
http://www.smarthouse.com.au/Home_Office/Storage/U9P4F7L2 [smarthouse.com.au]
for some years: dvd + raid-1 (Score:5, Informative)
Currently, There is no better way than store a backup on DVD and store the main data on a raid-1 disk set. Move the raid disk set to new disks every few years.
All the other technologies are more expensive, and even possibly more dangerous (loss of data due incompatibilies or for any other reason).
Recordable CD reliablity (Score:5, Informative)
Depends on the manufacturer and dye formulation. Some have failed in as short of a time as eight months while others are good after nearly ten years. For very important stuff, it is far too risky to be relying on the manufacturer. It's probably safer to make it a habit of regularly make multiple backups your data.
Storage array. (Score:5, Informative)
Every couple of years, you can add an extra couple of drives. With drive capacities increasing as fast as they are, cost shouldn't be a huge issue.
Use backups (Score:5, Informative)
Regardless of the methods you choose, I would highly recommend using at least two different media.
If these videos are important enough to be stored for 10 to 20 years, then they are important enough to be backed up - it is always difficult to foresee long term failures in any technology. If you read the article on tin whiskers [slashdot.org] they mentioned that some failures can not be tested using short time span methods.
Re:Tape (Score:2, Informative)
Re:Wow, that's a lot. (Score:5, Informative)
I have not been able to find anybody willing to press discs in quantities lower than about 150. Pressing discs in quantities of, say, 5 would be a nice service for archival purposes, but it would presumably be expensive, since I think the setup costs for pressing discs are high.
Re:CDs still a pain. Keep it alive and available. (Score:3, Informative)
Newegg has Quantum DLT SATA drives (160GB native capacity, 35 GB/h throughput) for about $700, so it won't break the bank to get proven multi-decade shelf-life media of reasonably size and speed for a 100GB dataset.
Every real OS has tape backup support (though you may have to hunt for drivers). If you're stuck with Windows, type ntbackup at the command line - it doesn't suck for home use.
Re:My method (Score:3, Informative)
Yea, never dismiss the old standard of posting it online and making storage someone else's problem.
But yea, if you are serious about storing, store on HDD, and keep an offsite backup. If you're careful with your offsites (i.e. you make a new FULL backup on a semi-regular basis), you can use DVDs, but like everyone else has already said, optical media is a crapshoot, and if you depend on it, you can depend on it letting you down.
Considering that you're still under a TB, I'd invest in a pair of externals, and switch 'em back and forth to your offsite every 6 months or so. (That sounds complex, but we're really just talking about leaving one at Mom's on Easter and switching it out at X-mas or whatever).
Re:HDDs (Score:3, Informative)
Better yet, buy two of them and mirror. Use some good filesystem -- like ZFS (available in Mac OS X, OpenSolaris, FreeBSD and to some extent in Linux), which will detect corruption and heal files using redundant copies. (google "zfs self-healing" for more)
Re:Recordable CD reliablity (Score:3, Informative)
NARA and Library of Congress (Score:5, Informative)
Forget Wikipedia, ask the people who spend their lives trying to figure this out.
http://www.digitalpreservation.gov/you/digitalmemories.html [digitalpreservation.gov]
http://www.archives.gov/preservation/technical/guidelines.html [archives.gov]
http://www.archives.gov/preservation/family-archives/digitizing-photos.html [archives.gov]
http://www.archives.gov/preservation/storage/ [archives.gov]
Re:Where can I get these mythical disks? (Score:5, Informative)
Re:CDs are still readable (Score:4, Informative)
CD-Rs are not the same thing as stamped CDs. With CD-Rs you're lucky if they last 5 years. Stamped CDs, if taken care of, will last practically forever.
Re:HDDs (Score:5, Informative)
Do not buy the largest hard drives that you can find. Cutting-edge storage densities might mean bad long term reliability. Go for something a little smaller than the leading edge that has had enough test time for you to find a large number of reviews on the drive's reliability. These will also be cheaper so you can buy two and store them at separate locations.
When you store your files, write a small utility or script that runs through them all and builds redundancy data, like QuickPar, and that stores a hash, e.g. MD5, so that 20 years later you can check the data is still good bit for bit, and even if it contains errors theres a high probability that you'll be able to correct it, even from the redundancy data stored on the same drive, let alone your second copy.
One important thing: As well as all of this, on each drive store a copy of the software (e.g. codecs), as well as any registration information to make them work. Backups of your files are no good if you can't play them later. Try to avoid storing video long term in any format that requires a codec with online activation. Will that company still be around in 20 years time?
I'd go DVDs, but definately not Magnetics (Score:2, Informative)
Re:CDs are still readable (Score:3, Informative)
More accurate would be "CD content still readable". CD rot is only an issue if you made too few copies and didn't re-copy to newer media periodically.
If it's really, really important:
--Save the stuff on reasonably reliable, name brand media. Make more than one copy, saved separately.
--Make the secondary copies on a different brand media, just to cover your ass.
--Copy over to identical media type a year or two later ( save originals )
--Copy/consolidate to the newer, cheaper media ( like CDs to DVDs) when the price on the new media drops a bit. Include drivers, codecs or even a player or two on each piece of media that consolidated several smaller pieces of media ( 5 CDs per DVD for example ). (Save originals )
--Repeat the re-copy to same type of media, repeat the consolidation to newer/larger media.
If original format is getting old or unusual, convert/transcode to newer format in the least lossy format. Save transcoded copies in addition to the originals ( don't throw away your negatives )
With the advent of consumer digital video/audio there's no reason to lose anything. Even saving all these CDs, DVDs, Blu-Ray disks would take less physical space than my Dad's box of 35mm slides that cover the same 20 year period.
Re:Diversify. (Score:2, Informative)
Re:HD unreliable (Score:4, Informative)
I was under the impression that unspun drives tend to deteriorate relatively quickly
Quickly? No.
- the heads clashing with the platter or some such nonsense.
Head crashes have been a non-issue since the late 1980s.
The real problem is the lubricant in the tiny motors. It can get gummy, and then the read arm won't move.
Just spin them up once a month and you're fine, from what I've heard.
I'd say bi-annually.
SSDs!! (Score:2, Informative)
Has anyone mentioned Key drives or SSDs yet? Technically those items are limited by the amount of writes you can do to them, which is sometimes an obscene number. So far, the key drive I had for over 5 years is still working, even though its been run over by a car.
Thoughts?
Re:Good luck with that! (Score:3, Informative)
Re:CDs still a pain. Keep it alive and available. (Score:3, Informative)
Raid is good for keeping the data alive, it is a backup in the sense of avoiding failure of devices causing major data loss. Raid 10 or a Mirror are your best bets for redundancy. However, a RAID is not going to be a preventative measure against other forms of data corruption, virus, batch file run amuck, accidentally deleting a folder. So you need a separate storage medium, you can pick your favorite, everyone has ones that work well for them. There are several alternatives depending on the size of data you are backing up. For me, I use online storage through a respected vendor MOZY for my home use, which kicks the ass of any other back up medium I have found, including tape. Tape is a dying backup medium in my opinion. I think what most people will be using within the decade are personalized net storage solutions.
BTW if you worked for me you would be fired for being an ass and pretending to know what you are talking about.
Re:CDs are still readable (Score:3, Informative)
I have the opposite experience (Score:5, Informative)
Your experience is very interesting because mine is the opposite. I make (at least) quarterly backups of my data and have since mid 1993 (CDs since 98, floppy before that). This spring I got bitten by the curisoity bug and started going through all my old backups looking for forgotten and interesting things. Every CD older than two years had at least one unrecovereable read error. Every CD older than five years, except for one, was completely unreadable. Between two and five years the number of read errors grew with many files being lost and several CDs being unusable. The 3-1/2 floppies were all 100% readable.
In that time period I've been through probably a dozen CD burners, both expensive varieties and cheap ones, and I've used at least as many brands of media. All the CDs have been kept stored in dark, dry, clean places. I tried reading the "unreadable" CDs on multiple computers and met some limited success accessing additional data. I didn't try any recovery software.
Fortunately for me I kept most of these backups out of habit and I didn't really care about much of the older ones outside of curiosity.
Re:That's one of the problems with Linux people.. (Score:3, Informative)
You could always use FreeNAS [freenas.org], which is slightly less complicated and can fit on a USB Key.
Re:CDs are still readable (Score:2, Informative)
Re:Recordable CD reliablity (Score:3, Informative)
I've had good luck ordering Taiyo Yuden media online, and the prices haven't been so bad, even in Canada. So far, I have had no trouble with them (but as we well know, that doesn't really mean much). I tried picking up some MAM-A recordable media, but it was a bit pricey for my needs.
NONSENSE! (Score:2, Informative)
I have CDs from back in 1998, when I've coaxed my boss to buy a 4x TraxData burner.
They still mount better and copy and open easier than some printed ones I got with various magazines over the years.
Aaah... but back then - a writable CD was about 5$ apiece and they only did up to 4x as did the writers.
I've also had (and still have) a large number of silverbacks burned by various people over the years.
Some of them were unreadable or had problems mounting the moment I've put them in the drive.
Just because it says somewhere on the sticker that it is 52X compatible or capable - it does not mean that it is.
That rule works for both disks and drives.
Also, whether it is cutting costs in manufacturing, packaging (I've found fingerprints on some "fresh" disks), quality control, transport or just plain lying about the performance - the ultra-cheap ones are cheap for a reason.
Re:CDs are still readable (Score:5, Informative)
Re:CDs are still readable (Score:3, Informative)
YMMV.
I have CDs from 1998 burned with SCSI cd burner and used cheap $1.00/CD bought in bulk that still read today.
Re: You're looking for the wrong thing.. (Score:5, Informative)
Wow, a misspelled spelling flame. It's like digging into the ground and pulling up a USENET post from 1993. Let me introduce you to a 21st century technology that might help you in future: it's called a search engine.
It's Taiyo Yuden, according to their own web site [t-yuden.com].
What are film & video archives doing? (Score:4, Informative)
AMPAS recently had a report called "The Digital Dilemma", which the NY Times wrote about [nytimes.com]:
If not operated occasionally, a hard drive will freeze up in as little as two years. Similarly, DVDs tend to degrade: according to the report, only half of a collection of disks can be expected to last for 15 years...
What are film archives doing? Where possible, studios are making long-lasting, non-fade B&W pan separation YCM polyester negative film backups, even when the film is mostly or totally "born digital". Then you put it under a mountain somewhere.
Government video archives worldwide are moving to LTO tape, typically using JPEG 2000 video encoding, with the recognition that every few years they will have to migrate their tapes up a generation of LTO. I suspect there may be a move from lossy JPEG 2000 to lossless JPEG 2000 and eventually uncompressed video as tape speeds and capacities ramp up.
Re:CDs are still readable (Score:5, Informative)
Ok, I just went looking for the handbook I got with my photography course, there's a paragraph about optical media: The CD's with a greenish look are guaranteed up to survive for 1-3 years. The siverish CD's last about 10 years. And there are also more expensive CD's with a gold color, and a black protection layer on top, that last +/- 100 years.
It also mentions there is no durability data about DVD's yet. This seems strange to me, and it's maybe outdated.
It might be wise to get some advice at a photography store, I'm sure they get a lot of those questions.
Re:Flash Storage is NOT GOOD (Score:4, Informative)
I'm amazed that no one mentioned it. Just get 16gb usb flash disks. It has theoretically unlimited life for archiving. The only time it deteriorate is when you continuously write/erase it.
Re:Flash Storage (Score:2, Informative)
Flash drive cells are constructed in the same way as dynamic memory cells, but with longer fade times. While they may last for a few years, they fade far, far faster than magnetic platters fail to entropy or the mylar in optical disks fails to radiation.
LTO has a good shelf life... (Score:2, Informative)
Battle Against Any RAID Five (Score:3, Informative)
Two things to think about:
RAID 5 suffers catastrophic failure when two drives fail within a few hours of each other [baarf.com]. The stress of rebuilding the replaced drive from the working ones makes this more likely to happen than some storage vendors want to admit. RAID 6 should do better because three failures in a short time are a lot less likely.
As far as I know, hardware RAID is risky if a RAID controller fails because one brand of RAID controller often can't read an array created on another brand. Software RAID on a Free operating system is more portable.
Re:CDs are still readable (Score:3, Informative)
CDs and DVDs are just not suitable for the task. They're costly, slow, difficult to verify and have a short shelf life.
Really, the way to go is hard disks. Get like 4 of them, set them up to be mirrored into 4 identical drives.
Then separate the two pairs, periodically pull them and verify that all the files still match their original checksum and replace the ones as needed.
You're not likely to find an option which is that inexpensive and reliable. Definitely not with optical media of any sort.
It's not perfect, but something very much along those lines is going to be the best bet for most people.
You can relatively easily switch them to newer disks as time goes on, and whenever a new controller technology comes out, you can easily buy a add in card and transfer them like that.
Re:CDs are still readable (Score:2, Informative)
The most memorable time I saw it explained was when I watched this 20 minute video which will help you think about your effect on the planet:
http://www.storyofstuff.com/ [storyofstuff.com]
Comment removed (Score:5, Informative)
Cost-effective for technical people... (Score:1, Informative)
This is what I've been doing for years and it has proven robust and manageable. Someone ought to productize this sort of solution to sell to non-technical people... :-)
1. Obtain two geographically separated sites with basic broadband, e.g. your house and some friendly relative or friend's house
2. Put together two cheap headless PCs with basic all-integrated motherboards and at least 4 internal SATA ports and drive bays. Total cost fo r the (diskless) system should be easily less than $250 US. (Use an existing monitor and keyboard for initial setup.) Ideally, add a basic home/small-offic UPS that can be monitored by the Linux host for clean emergency shutdowns.
3. Choose your disk sizes for best value for RAID 5 with about 250% more space than your expected archive size. I use RAID 5 with three disks and have the option of rearranging disks later for expansion. 1 TB via 3 500 GB drives would cost less than $250 per host.
4. Install Linux and use software RAID 5 so the drives can be moved to new machines etc if there are failures.
5. Create a regular filesystem and a "backups" filesystem as separate RAID 5 arrays on the same machine. (Each disk has several partitions, one being a part of each RAID 5 volume.) Run nightly incremental-generational backups between local volumes via cron jobs. E.g. rsync with --link-dest mode can build link trees of shared data and only use significant space when new unique files are added.
6. Mirror the regular filesystem contents between the two machines once via high-speed LAN. Both machines continue to make local backups of their own regular filesystem.
7. Move the slave mirror machine to the secondary site. Leave both machines running 24x7.
8. Run regular mirroring from the master to the slave machine via cron, e.g. use an appropriate rsync+ssh job. Use dynamic DNS to establish the connections easily.
9. Enable SMART monitoring on both machines and actually check them regularly.
This solution protects against localized disasters that destroy an entire host. It also protects against simple disk failures, with less downtime or recovery effort than a total machine loss. The backup system protects against user error or corruption of the regular filesystem by keeping generational copies of older data.
The use of rsync makes it viable to sync frequently over broadband connections. Whether you can afford nightly or only weekly depends on your broadband speeds and rate of change of data.
These servers cannot be left alone for decades, but eventually need to have the data read/write scrubbed due to the limits of magnetic storage.
But, in a practical environment with new data being added, you will probably find that it is time to replace failed disks or increase storage space long before this issue arises.
If you strategically replace drives with equal size or double size drives, depending on cost and projected storage needs, you can eventually migrate to larger filesystems once you upgrade enough disks. For example, on a shoestring budget you can start with 3 x 500 GB disks; replace failures with 1 TB disks; eventually grow your arrays when you have either 3 x 1 TB disks or 2 x 1 TB + 2 x 500 GB installed. You will have a situation where your existing 1 TB usable space is only using the first half of each 1 TB drive (and one of the two 500 GB drives in the mixed size case).
Because of the dual-volume approach for regular and backup filesystems, you can initialize the new space on the second half of each 1 TB drive, copy all data into it, then reformat the old space when converting from a 1 TB to a 2 TB usable space. Also, because of the multiple volumes, you can us ea mixture of 500 GB volumes from "half" a 1 TB drive and a complete 500 GB disk.
Re:Diversify. (Score:2, Informative)