Long-Term Personal Data Storage? 669
BeanBagKing writes "Yesterday I set out in search of a way to store my documents, videos, and pictures for a long time without worrying about them. This is stuff that I may not care about for years, I don't care where it is, or if it's immediately available, so long as when I do decide to get it, it's there. What did I come up with? Nothing. Hard Drives can fail or degrade. CD's and DVD's I've read have the same problem over long periods of time. I'd rather not pay yearly rent on a server or backup/storage solution. I could start my own server, but that goes back to the issue of hard drives failing, not to mention cost. Tape backups aren't common for personal backups, making far-future retrieval possibly difficult, not to mention the low storage capacity of tape drives. I've thought about buying a bunch of 4GB thumb drives; I've had some of those for years and even sent a few through washers and driers and had the data survive. Do you have any suggestions? My requirements are simple: It must be stable, lasting for decades if possible, and must be as inexpensive as possible. I'm not looking to start my own national archive; I have less than 500GBs and only save things important to me."
Quality DVDs, archival storage, repeated backups. (Score:4, Informative)
Long term:
Use quality DVDs. Redo the backup on a schedule such that everything is re-backed up every three years or so. Every month, say, you make one DVD. Keep the backups in a climate controlled, dark, secure place, such as a safe deposit box at the bank.
Short term:
Back up everything you want to save to an external hard drive weekly. Every three months swap it with a drive kept in the safe deposit box.
Daily:
If you have a Mac, use Time Machine. If Linux, some sort of cron job running a Python script that copies /home to an external hard drive. If Windows, I dunno.
Ask Slashdot AGAIN (Score:5, Informative)
As for the question itself, the answer is pretty simple, but unhelpful. Basically what it comes down to is that there is no safe place for your data. You're asking for the best type of basket to put all your eggs in. If you look at it that way, the solution is easier to arrive at. Your choices are A) spare no expense and build/buy the world's strongest basket and pray no flaw arises, or B) start copying your eggs around to all sorts of cheap baskets and continuously add more baskets in the expectation that the oldest baskets are going to fail.
Copy all your stuff to all your computers. Burn to DVD and/or CD ROM. Buy SD cards and USB flash drives. High capacity storage devices are so cheap now that you can keep all your valuable pictures of your vacation to Cleveland quite safe by constant duplication. That's the value of digital. Copies are perfect. Make lots.
Wrong question (Score:4, Informative)
It's impossible for guarantee 100% storage integrity, just like it's impossible to guarantee 100% uptime. What you want to ask is what risk of data loss you are willing to take.
This page compares some of the options [sun.com] in terms of Mean Time To Data Loss (MTTDL). For the amount of space you're looking at (~500gb), a three-way mirror is probably sufficient to last for your lifetime.
But there's always the risk of fat-fingering "rm -rf" or having the building catch fire, so maybe you want to have two synchronized sets of mirrors, stored in different physical locations. Only you can decide if that's too paranoid for you (or not paranoid enough).
Jesus saves, but Buddha makes incremental backups. (Score:2, Informative)
But seriously, I've had the same (but growing) data set in my /home for over 15 years, and going. I find the easiest way is to just keep it on my drive, and have a few frequently updated copies on external media (optical or solid or dirve) and to keep it on another PC too, disk space is so goddamn cheap. I also have a large music collection, and instead of wasting time backing it up onto optical media, I just keep it on both my notebook and PC, its unlikely both will fail at the same time, and incase of a robbery, I can also archive it at work.
Don't expect any form of media to last forever, it's multiple, frequently updated copies that will ensure your data lasts forever.
Also, if you have friends and family you can trust, make a copy for them to keep for you, off-site backup is also important.
Obviously this all depends on how important and/or private the data is.
my 0.2
Re:Amazon S3 (Score:5, Informative)
Its not really that cheap, and not that simple to use for personal backups. Unless you are willing to write your own backup scripts, its going to be a headache.
Querying S3 for a list of stored files is *very* slow, and you only get 1k results per query. This means you have to index what files you put in S3 in a local db. This allows you to ask the db what files are there (and how to grab them).
If you only have a few files you can use the S3 browser extension for Firefox (or one of a many file system mounting, ftp simulating, etc tools). Just keep in mind the 1k file limit per query and box things in folders of no more than 1k items. Otherwise you will have a very slow browsing experience.
I have around 120 GB of family photos and purchased mp3s that I would like to store. To store 120 GB at .15 per gigabyte/month for 1 year would cost me: $216 (at $18 a month).
We use it where I work, with great success, but it would be much to much work for me for a personal backup system.
Considering the cost, I would go with a consumer targeted app (there are LOTS of them). A number of them charge a flat flee for "unlimited" storage. Beware of how you interface them. Some support windows only.
Re:Amazon S3 (Score:5, Informative)
That is an interesting suggestion, but 500GB would cost $900/year (plus transfer costs) which I don't call "dirt cheap". As far as being there forever... who knows if there will be an Amazon in 10 years? Amazon might be more stable than most hosting options, but forever is a long time.
Again, Quality DVDs (Score:2, Informative)
Re:Not enough history (Score:5, Informative)
The spindle bearings can seize up.
The problem isn't that the drive will inevitably die after 5 years, it's that it won't inevitably last longer.
Re:Hard drives kept online (Score:5, Informative)
I recently built my own cheap backup server using OpenSolaris and ZFS. I used my old SATA drives (6x400GB), a $75 motherboard and AMD Athlon X2 combo, 4GB of DRAM ($69) and an old tower case. I did add two SATA 5-bay hot-swappable disk bays ($110 each) so that I can easily replace/upgrade my disks. Once a week I update data from my main server (also Solaris) to the backup server using ZFS incremental snapshots.
My PC's and Mac's all mount their user directory from my main server, and I rsync my laptop every day. The main server also serves as a SunRay server so I do most of my daily chores on a SunRay. I run Windows inside VirtualBox and I rarely ever turn on my windows PC anymore (the Windows instance in VBox also mounts from my main server). Inside my main server I have 2x 1TB drives, in a ZFS mirror setup, for the user directories and 2x400GB for the OS and scratch directories (all drives are SATA).
I'm very confident in this setup, also because I can yank out my drives in under 30 seconds in case of fire. The only thing I still have to do is put my backup server in a different room from the main server - that is a todo project for the near future.
Stop asking Slashdot about this... (Score:3, Informative)
... honestly, Slashdot - and others - have covered this time and time again. Nothing has changed. There still isn't a cheap digital storage medium that we know for sure -will- retain your data -and- be readable (in terms of media -and- the hardware to read that media) down to the very last bit for your grandchildren.
IF and when there's a breakthrough, I'm sure Slashdot is one of the first places you'll hear about it.. but it won't be in an answer to an Ask Slashdot.
http://it.slashdot.org/article.pl?sid=07/04/21/1257249 [slashdot.org] - Digital Media Archiving Challenges Hollywood .. and so forth and so on.
http://hardware.slashdot.org/article.pl?sid=06/11/20/2036247 [slashdot.org] - Archiving Digital Data an Unsolved Problem
http://hardware.slashdot.org/hardware/06/12/11/1714232.shtml [slashdot.org] - How To Choose Archival CD/DVD Media
http://hardware.slashdot.org/article.pl?sid=05/06/26/218250&from=rss [slashdot.org] - Archiving Digital History at the NARA
http://tech.slashdot.org/article.pl?sid=07/03/31/2141204 [slashdot.org] - How To Properly Archive Data On Disc Media
Yes, I realize that you stated "I'm not looking to start my own national archive; I have less than 500GBs and only save things important to me". However, it doesn't really matter whether you're archiving hollywood movies, NASA records or just your own random crap. If it is important to you - important enough that you want it to be "lasting for decades if possible" - then your concerns are the same as NASA's... and they're struggling with the exact same question.
The 'best' answer so far is one you will find in each and every single discussion on this - including this thread, so I'll just point you there:
http://ask.slashdot.org/comments.pl?sid=1061489&cid=26102825 [slashdot.org]
You mentioned 'cheap', as otherwise all the answers saying "duuuuude, ditch the digital - go analog!" might have some validity.. take a wild guess as to what it would cost to have thousands of photos transferred to negatives/prints, or video transferred to tape/film, etc. Plus you mentioned documents.. some of those may not transfer to e.g. paper (easily) at all depending on the 'documents' in question; e.g. CAD files.
Re:Amazon S3 (Score:4, Informative)
Jungledisk [jungledisk.com] takes care of all the tedious backup stuff for you, and it is only a one time charge for the app.
But you're right, S3 isn't cheap. To store 500 GB of data would be about $75 a month, plus the $50 to put it on the server in the first place.
Re:Hard drives kept online (Score:5, Informative)
I recently built my own cheap backup server using OpenSolaris and ZFS. I used my old SATA drives (6x400GB), a $75 motherboard and AMD Athlon X2 combo, 4GB of DRAM ($69) and an old tower case. I did add two SATA 5-bay hot-swappable disk bays ($110 each) so that I can easily replace/upgrade my disks. Once a week I update data from my main server (also Solaris) to the backup server using ZFS incremental snapshots.
My PC's and Mac's all mount their user directory from my main server, and I rsync my laptop every day. The main server also serves as a SunRay server so I do most of my daily chores on a SunRay. I run Windows inside VirtualBox and I rarely ever turn on my windows PC anymore (the Windows instance in VBox also mounts from my main server). Inside my main server I have 2x 1TB drives, in a ZFS mirror setup, for the user directories and 2x400GB for the OS and scratch directories (all drives are SATA).
I'm very confident in this setup, also because I can yank out my drives in under 30 seconds in case of fire. The only thing I still have to do is put my backup server in a different room from the main server - that is a todo project for the near future.
Problem 1: If you are not home and your power supply decides to catch fire, you have lost everything.
Problem 2: If you are home, you better be spending those 30 seconds trying to get your butt out of the fire, not running after hard drives.
If you think your DR plan relies on yanking drives out, you're in serious trouble. One B&E or a fire and your data is gone. Now this may be perfectly acceptable to you. It is to a lot of small companies, until it happens to them.
Personally, I vault offsite on a daily basis as well.
Re:Not enough history (Score:3, Informative)
Bit Rot [wikipedia.org]
Re:Flash drives (Score:3, Informative)
Actually Flash/memory drives are sensitive to radiation. Long term storage without regularly accessing the drive can lead to situations where blocks go bad beyond the ECC/CRC capabilities of the drive to fix. If you intend to store valuable data on memory devices for the long term you should (a) use multiple redundant drives (b) use a file-system with block-level ECC/CRC error correction and redundancy (like ZFS) (c) write each block to the device twice in different location (i.e. an mirror on the drive).
The future of Flash memory is such that unless they extend the ECC/CRC capabilities of the controller, the susceptibility of these devices for radiation will increase when the cells get smaller.
In case anybody doubts the impact of radiation on electronic devices, here is an interesting experiment you can do: take your digital camera, put the lens cap on and do timed exposure with increasing exposure times (1,2,4,8, ... seconds). Then analyse these pictures for bad-pixels, or better, subtract the pictures from each other. The random bits scattered around on these frames are impacts of cosmic rays. Now apply the same principle on memory devices with much longer exposure times...
To cut my somewhat rambling post short: use memory devices as long term storage? No. Not without thought about the required data reliability.
Re:Magnetic Tapes... (Score:4, Informative)
So start building VM's of operating systems and software that are in use. Archive those. Far from perfect or complete, but it should narrow the scope of the problem a little bit.
As far as personal stuff, I think the best solution is to have 2 or more live copies of all important data and just migrate them to whatever makes the most sense at a given point of time, and then also have backups of stuff. That doesn't work with the question, but there isn't really a cheap answer to the question at this point.
Re:Hard drives kept online (Score:5, Informative)
I'm not sure what "striction" is (and I suspect I don't really want to find out). "Stiction" on the other hand, refers (among other things, check out the Wikipedia link) to the tendency of the lubricants inside the HDD to increase in viscosity over time. After a long enough period of time, the motor can't rotate the platters. The fun thing about this problem is that the 'repair' involves brute force - banging the HDD to move things a bit.
Supposedly was a problem on older drives. My brief Google search did not yield any useful info. The problem with HDDs is that they are designed to be on and NOT designed for long term, powered off storage. That said, turning your USB drive on and off shouldn't be much of an issue. Remember the thing was probably made 6 months before it was installed in your housing and then sat in a warehouse for a while. The question is - can you expect a random HDD to fire up after sitting in storage for x years (x being larger than 2 or 3)?
Not Amazon S3 (Score:5, Informative)
Re:Hard drives kept online (Score:5, Informative)
Re:Amazon S3 (Score:2, Informative)
What the OP needs to understand is that someone has to do periodic maintenance over those decades. Either he does it himself or hires someone (either directly or as subscription to a service). All archiving requires monitoring of the storage media and preventative transfer to newer media before the information is lost. Given that media is not perfectly predictable, you need redundancy and frequent monitoring too, so you can discover a data loss and restore redundancy before all media sets are lost.
Using flash/solid state drives is probably going to improve the shelf-life of a storage copy. But you still have to monitor for loss. It doesn't reduce the chances of theft, fire, flood or simple misplacement.
I do it myself, with cheap Linux software RAID boxes in separate locations, piggybacking on existing Internet services to do rsync mirroring between the sites. The machines are kept alive, running disk scans. The rsync mirror process is also periodically accessing the files at application level. I may add some cron job to do checksum verification some day.
The maintenance cost is the power supply, internet service, eventual hard disk replacements (for failure or size increase) and my time. It is the cheapest solution for me that has the level of reliability I can understand and control myself. It is affordable because I have a friendly site letting me colo my remote box for free/good will. I don't pay their internet or electricity bill. I am geeky enough to consider the several hours per year of effort to be part of my computer hobby.
My solution has evolved to this point and run stable for the past 5 years or so. Prior to that I did similar concepts but with less reliable equipment for each mirror, e.g. depending on fileservers at school and work that I could piggyback on with a few tar files, and just using single-disk machines locally. I've propagated my important date in "online" form for about 15 years now. Anything from before that was floppy disks and is lost to me, but also doesn't matter because we're getting back to teenage years by then.
Re:Hard drives kept online (Score:5, Informative)
I tend to agree, however I'm a bit confused over what exactly is being requested.
"I've thought about buying a bunch of 4GB thumb drives....I have less than 500GBs and only save things important to me."
At first glance I thought you had 500gb you were trying to store, but then you mentioned "buying a bunch of 4GB thumbdrives" and I can't imagine someone buying 125 4gb thumbdrives to use for backup. So exactly how much data are you trying to store?
If less than 50gb, I'd suggest a few SD cards. 8gb SD is ~$11 [amazon.com], or 16gb for $30 [amazon.com]. While more expensive than hard drives per gb, SD cards are remarkably resilient, surviving a week in the ocean [letsgodigital.org], and a few in a ziplock bag stored in a safe deposit box would probably last close to forever.
SD will probably still be around at least for the next decade or longer. SD has already been around since 1999 [wikipedia.org] and all modern card readers read SD cards by default. SD slots are in nearly every form of consumer electronic device, and every manufacture of digital cameras uses SD except Sony and Olympus [wikipedia.org], almost guaranteeing the card readers will be around for many years to come.
I would suggest against USB anything since they're already discussing cutting the cord on USB and going wireless USB [intel.com]. While I don't predict that will happen overnight you wanted a solution that would be available decades from now, and wired USB might go the way of the parallel port [wikipedia.org], which was the standard external port in the 80s and 90s but was replaced by USB late 90s. Parallel port only had a lifespan of about 20 yrs and is no longer on modern PCs, and USB has been out just over 10 years [wikipedia.org] so it's feasible in 10 years PCs will no longer have USB ports, everything will be wireless USB.
Re:Amazon S3 - until? (Score:3, Informative)
Yeah - sure - until Amazon goes out of business or gets bought and then the new owner dumps the service and you're S.O.L.
RS
Re:Hard drives kept online (Score:3, Informative)
Really? Taken apart any modern hard drives lately? "Parking" is simply the act of moving the heads to an area of the platters not used for data storage. The heads are still very much in contact with the platters, but if the heads get bounced around, there's no data under them to damage.
Stiction is caused by centripetal force of the spinning platters gradually drawing spindle bearing lubricant across the surface of the platters. This happens gradually over a long period of time. Once that happens, if you leave the drive powered off (parked or not) for a significant period of time, the contaminants can bond the heads to the platters.
Newer drives take steps to prevent this such as using better bearings and parking the heads close to the spindle so they can generate more "break-free" force from for a given amount of motor torque.
If you copy data to a brand new drive (preferably a lower capacity unit with a single platter and only a pair of heads) and then take it offline and store it in a climate controlled environment, stiction is not likely to ever become a problem.
Re:Hard drives kept online (Score:2, Informative)