Long-Term Personal Data Storage? 669
BeanBagKing writes "Yesterday I set out in search of a way to store my documents, videos, and pictures for a long time without worrying about them. This is stuff that I may not care about for years, I don't care where it is, or if it's immediately available, so long as when I do decide to get it, it's there. What did I come up with? Nothing. Hard Drives can fail or degrade. CD's and DVD's I've read have the same problem over long periods of time. I'd rather not pay yearly rent on a server or backup/storage solution. I could start my own server, but that goes back to the issue of hard drives failing, not to mention cost. Tape backups aren't common for personal backups, making far-future retrieval possibly difficult, not to mention the low storage capacity of tape drives. I've thought about buying a bunch of 4GB thumb drives; I've had some of those for years and even sent a few through washers and driers and had the data survive. Do you have any suggestions? My requirements are simple: It must be stable, lasting for decades if possible, and must be as inexpensive as possible. I'm not looking to start my own national archive; I have less than 500GBs and only save things important to me."
Gmail (Score:3, Interesting)
Sorry, it's insoluble. (Score:5, Interesting)
I believe this to be a serious problem with no good solution currently. That's the truth. You'll get lots of dismissive posts saying it's no big deal, but it is.
Forget media integrity. The problem is technology drift. Everyone thinks "ubiquitous" (as in every computer has a USB port) is the same as "eternal," and it isn't. Twenty years from now, your USB thumb drives and CD-R's may have their data physically intact, but only museums will have equipment that can read them.
It is a fantasy to suppose that you can successfully perform Sisyphus-like task of systematically recopying your data to new media and formats. The proof of this is the innumerable stories of big, well-funded organizations that have neglected to do this. If the NASAs of the world keep finding reels of tape with important data on it that can't be read due to technology skew, what makes you think that you can do much better?
(What makes me bitter is failure of vendors to give adequate warning when software updates remove the capabilities of reading file formats that were formerly supported. I once verified that my new Mac could read my old MFS diskettes, and did not notice when a software update to the OS removed that capability. Microsoft was less than forthcoming when they removed the built-in ability of Excel to read Multiplan files).
Re:Magnetic Tapes... (Score:5, Interesting)
Having a tape around for 20 years doesn't do you any good. 20 years ago, I was writing to 1600 and 6250bpi tapes. Today, my data center doesn't even have a drive that can physically read them.
Today's tape technology is no different. 3 years ago was writing to SDLT tapes. By next year, I won't even have an SDLT drive in my data center, having migrated everything over to LTO.
Yeah, I have round tapes in my offsite storage. I have 4mm and DAT tapes out there. We're just wasting money storing the media, since we have nothing that can read them.
If I could read the old media and extract a really old database, would today's database app be able to read it? Probably not. And could I install that app on today's OS? Probably not. And could I install the OS from many years ago on today's hardware? Probably not. Could I compile source from 20 years ago with today's compilers? In many cases, actually I can't. And if it really did all magically get compiled, is anybody around that can still knows how to run the app?
Don't forget that 20 years ago, many systems didn't have TCP/IP installed. In 1988, mine didn't - it was a combination of RS232-attached terminals and XNS-attached graphics workstations. Drive sizes were 80-160MB. A couple of MB of memory was a lot.
For those of you not still in school, ask around and see how many folks in your IT department can name the server that held your financial data 10 years ago.
Re:Flash drives (Score:4, Interesting)
By the way, I *have* had an SD card fail. It was in my digital camera the whole time, worked fine for a couple years, then quit. The camera itself showed no sign of damage, so I don't think it was abused. It was a Kingston, too, which I consider reputable.
Re:Hard drives kept online (Score:3, Interesting)
[citation needed], please. Really, my understanding is that non powered hard drives are NOT good long term candidates due to 'stiction' (maybe not so much an issue now, maybe it is) and perhaps other problems.
(Warning: Anecdote time) I've had several previously good HDs in my junk box fail to start when I decided I wanted to play with them again. Any real data?
Re:Not the media that's the problem (Score:5, Interesting)
There really is a simple way around this - and it is what I've done - I've got data 25 years old and it's still relatively easily manipulated with a little work. I've found floppy disks are relatively resilient, and old hard drives seem to keep their data for a long time. I've got a computer, display, keyboard, and associated peripherals stored for every generation of data that I kept:
1.I have a Commodore 64 with floppy drive and cassette drive stored in a box with the floppy disks and cassettes from that generation (late 70s/early 80s).
2.I have an IBM PC/XT with keyboard, a 5 1/4" floppy, 3 1/2" floppy, internal 20MB hard drive, and CGA monitor stored in a box with a load of 5 1/4" floppies filled with data from that generation (Mid 80s).
3.I have an IBM RS/6000 with display, keyboard, and mouse and internal 500MB hard drive loaded with all my docs and projects from that generation (early 90s).
4.I have a Pentium 2/300 PC * 15" monitor with windows 98, CD R/W drive, 3 1/2" floppy drive, and USB ports - and a crapload of CD's and 3 1/2" floppies full of stuff from that generation (Mid/late 90s).
When the current generation looks like it's going to be moving on, I'll put away a Core 2 Duo system with 1 TB of hard drive full of stuff with the different OS's I used loaded on it with boot manager (Ubuntu, XP, FreeBSD), a crapload of USB keys full of documents, along with burned DVDs etc. That'll take care of the "'00" generation.
The answer lies in not only archiving your data "of the generation" but the essential equipment needed to access it. I may have a heck of a time moving data off of my Commodore 64 - but I can at least see it and access it - I believe I stored a modem with it - so at worse I could set up a terminal server that it could dial into and dump data to. All the other systems I'm pretty sure I could recover stuff from - even if the PC/XT does have an MFM hard drive, etc.
Re:Amazon S3 (Score:1, Interesting)
Well it's not dirt cheap for 500GB - that's $75/month. But who really has 500GB of critical data.
If most of that 500GB are photo's then for $24 a year, one can get a pro account at flickr with unlimited storage and retrieval.
The rest can then be backed up to Amazon's S3 (which is only $0.15 a GB, very reasonable for small amounts)
Friends + rsync over ssh (Score:3, Interesting)
I have gigs of photos (wedding, long lost family picnics, etc) and music that I can't bare to lose.
Find someone who doesn't want to lose anything either and setup rsync over ssh. Synchronize often, rsync is very friendly to bandwidth.
Re:Hard drives kept online (Score:3, Interesting)
What's "striction"? I'm asking because I just switched-off my USB drive in order to save power. Maybe I should turn it back on & let it run continuously?
My suggestions:
- Home videos - nothing lasts as long as analog videotape. Twilight Zone episodes recorded in the 1960s are still viable today, whereas DVD-Rs have a nasty habit of self-erasing themselves in just 2-3 years. I think Super VHS tape is the safest way to store family memories.
- For files smaller than 20 megabytes, I upload them to yahoo mail and Gmail.
- For larger files I dual-duplicate them across both my c: and usb: drives. If one fails I know I'll still have the backup of the other.
- For source code, printing to paper is also an option. Good quality paper will outlive you.
-
- And finally: Ask yourself if you really need that stuff? Do you need to save your old CSE 101 project? Probably not. I used to save that junk but then I realized it was pointless. Nowadays I save very little because most of it is not worth keeping or will never be used again.
Re:Magnetic Tapes... (Score:3, Interesting)
At my work (18,000 desktop Nat'l research lab), we have a special group to keep old hardware around, just so they can go and pull data from old archives.
Re:Magnetic Tapes... (Score:3, Interesting)
You can definitely do that today with DOS and Windows 3.1 on todays hardware. If we move away from x86 architecture, then everything is out the window, but for now, yes, we can install OS from many years ago on todays hardware.
Tried that lately? Not sure about DOS and Win3.1, but certainly Win95 had a boatload of timing issues on newer hardware.
Re:Magnetic Tapes... (Score:0, Interesting)
to be perfectly honest people think their data is so important as to be wanted or needed in '20 years time'. Perhaps there is some data that might, but mostly it's just a digital form of hoarding, nothing more.
Just look around you and wonder what you don't have now that you wish you could get back from 20 years ago. If you're honest with yourself, I doubt there is much really, and what there is, is probably not appropriate for a digital world (an old favourite cuddly toy, or a sentimentally valuable lamp that was lost in the last move).
I'm always surprised for instance how much email people keep, my archived inbox is has never really been more than a couple hundred MBs. I've never in the 15 years or so I have had an email account (and moved/iported inboxes) ever regretted anything I have deleted or inadvertently lost.
Perhaps with the rise of MP3s and movies on line this might be more relevant, but I think not that much. People just buy it again. Think Vinyl, cassette tape, CD and now MP3.
It's mostly about ego or 'my backed up stuff is bigger than your backed up stuff' with people measuring their personal worth in the amount of bits they keep.
This is actually a good idea (Score:5, Interesting)
This actually is a good idea. If the porn files were maybe one half sex imagery and one half encrypted private data, and there was no easy way to separate the two halves, then people would download, store, and upload the files in order to view the porn. Anyone who had data in the private section of the file could download it from various P2P sites.
The cost of filming and creating the porn file would be covered by the people who would be using the file for long-term distributed storage. Say a 1.5Gigabyte file that was an hour of MP4 video entertainment and 500 megabytes of distributed storage. The fees received by the producers for the storage would pay for the video production costs. Since porn is cheap to produce, this may solve the problem of piracy and secure storage at the same time.
Re:Amazon S3 (Score:3, Interesting)
I have trouble with this in my mind, since so much of my work is devoted to making sure that information always exists and is accessible all the time. I look at these personal solutions for backup, and I'm so used to evaluating enterprise-type products that I scoff.
I guess that I don't know anymore what is appropriate for 'home users' when they say they want to keep data long-term. The submitter stated that tape drives were inadequate when that's still the most reliable method that enterprises use.
Sure, there are VTLs, but to not keep your data offsite as well would be counterproductive. I guess I just don't know what the submitter wants.
ZFS storage (Score:1, Interesting)
I have a system using FreeBSD servers running ZFS as the file system. The servers are built with around 6 disks each in a ZFS RaidZ2 pool. This allows each system to have upto 2 disks fail without losing any data. Data stored on ZFS is also internally checksumed which means that the files will stay 100% pure and not suffer from bit rot. I have 2 physical machines in different locations which are synced using rsync. Each machine runs smartd to run SMART checks on the hard drives weekly and they also run a zfs "scrub" weekly to verify that all of the data is in place and is correctly checksum verified.
For critical files (wedding photos etc.) I use par2 software to create a further checksum and generate additional redundancy. The par2 software allows corruption to be detected and corrected. This was more of an issue before I was running ZFS however it's still better to be safe then sorry.
Daily snapshots are taken of the data to protect against accidental deletes. Critical storage is moved to read-only zfs volumes.
The system has a few flaws, the first is that it doesn't protect against malicious damage. It would be possible for someone with admin access or an axe to "zfs destroy" my files. I have some critical fil2es dumped to DVD however I should probably have a third redundant "offline" site where all of the files are backed upto manually. Perhaps just a collection of removable drives.
I am also concerned about data file compatability. I currently store my digital camera files as RAW camera files. Critical images I also store as Adobe DNG files and as JPG's. One of these formats should be hopefully supported by Photoshop CS428 in the year 2100.
The system can periodically have the hard drives, pc hardware and software upgraded or replaced without disturbing the data so I can see no reason why the system can't keep the data valid for many many years.
One concern is that after I die and stop maintaining the system it may stop working. I guess it won't be my problem.
Re:Not enough history (Score:5, Interesting)
I let a lot of my early programs go. I had them on 9 track digital tape, at 6250 bpi. I had converted some of them from the lower bpi rates before. It used to be that each employer I worked for had one of those tape drives and I could access the files
when I wanted to. But not anymore. Besides, would I really want to convert FORTRAN programs?
In theory, I could have kept converting them to newer and newer storage media, but I didn't.
Later on, I had one of those QIC tape drives that could hook up to a floppy disk controller.
I think I have those files backed up on a hard disk somewhere -- I think. When I was going through old backups one year, I noticed I had the same directories appearing many many times.
I spent hours selecting the best version of everything and then backing up that too.
I haven't digitized all my vinyl record albums either, even though it is theoretically possible.
I have not scanned all the 35 mm slides or prints I took going back to the 1970s. At some point, the new stuff going on in your life gets more interesting than converting the stuff that used to get you excited 25 years ago.
Imagine how hard it is going to be to preserve people's papers after their dead. Do you want to keep converting someone else's stuff when you have a few terabytes of your own to attend to?
For now. (Score:3, Interesting)
There really isn't a good solution right now.
After thinking about it a while I realized that:
1) Most of the stuff on my computers could be replaced.
2) The one thing I would really, really hate to loose are family photo's.
3) Hard drives WILL fail sooner or later.
4) Tapes are reliable for a while but even in a climate controlled vault I have had tapes at work end up bad after a couple of years. (not to mention the pain in the neck it is to find a working legacy tape drive after 10 years)
5) DVD's will probably have the same issues.
My solution for now is redundancy.
Digital photos get offloaded to my Linux pc. I use a program called Digicam. .5T USB external disk drives with the "one button backup" program that is set to run nightly.
I have a bash script that syncs the new photos to a Windows share on my wifes pc.
My wife has one of those
When I have a couple of new directories of photos I run another script that compresses the whole directory and splits the output into a bunch of 45 megabyte rar archives.
I then upload them to Microsoft's free "Skydrive". Microsoft just upped their free disk storage to 25GB.
I also have some documents saved on the free AOL Xdrive.
I figure in a couple of years there will be a better long term storage option. It will probably be something like a solid state drive that lasts for two hundred years. At that point I will save everything to that and store it in my safety deposit box at my bank.