Long-Term Personal Data Storage? 669
BeanBagKing writes "Yesterday I set out in search of a way to store my documents, videos, and pictures for a long time without worrying about them. This is stuff that I may not care about for years, I don't care where it is, or if it's immediately available, so long as when I do decide to get it, it's there. What did I come up with? Nothing. Hard Drives can fail or degrade. CD's and DVD's I've read have the same problem over long periods of time. I'd rather not pay yearly rent on a server or backup/storage solution. I could start my own server, but that goes back to the issue of hard drives failing, not to mention cost. Tape backups aren't common for personal backups, making far-future retrieval possibly difficult, not to mention the low storage capacity of tape drives. I've thought about buying a bunch of 4GB thumb drives; I've had some of those for years and even sent a few through washers and driers and had the data survive. Do you have any suggestions? My requirements are simple: It must be stable, lasting for decades if possible, and must be as inexpensive as possible. I'm not looking to start my own national archive; I have less than 500GBs and only save things important to me."
Hard drives kept online (Score:2, Insightful)
Re:Hard drives kept online (Score:5, Informative)
I recently built my own cheap backup server using OpenSolaris and ZFS. I used my old SATA drives (6x400GB), a $75 motherboard and AMD Athlon X2 combo, 4GB of DRAM ($69) and an old tower case. I did add two SATA 5-bay hot-swappable disk bays ($110 each) so that I can easily replace/upgrade my disks. Once a week I update data from my main server (also Solaris) to the backup server using ZFS incremental snapshots.
My PC's and Mac's all mount their user directory from my main server, and I rsync my laptop every day. The main server also serves as a SunRay server so I do most of my daily chores on a SunRay. I run Windows inside VirtualBox and I rarely ever turn on my windows PC anymore (the Windows instance in VBox also mounts from my main server). Inside my main server I have 2x 1TB drives, in a ZFS mirror setup, for the user directories and 2x400GB for the OS and scratch directories (all drives are SATA).
I'm very confident in this setup, also because I can yank out my drives in under 30 seconds in case of fire. The only thing I still have to do is put my backup server in a different room from the main server - that is a todo project for the near future.
Re:Hard drives kept online (Score:5, Informative)
I recently built my own cheap backup server using OpenSolaris and ZFS. I used my old SATA drives (6x400GB), a $75 motherboard and AMD Athlon X2 combo, 4GB of DRAM ($69) and an old tower case. I did add two SATA 5-bay hot-swappable disk bays ($110 each) so that I can easily replace/upgrade my disks. Once a week I update data from my main server (also Solaris) to the backup server using ZFS incremental snapshots.
My PC's and Mac's all mount their user directory from my main server, and I rsync my laptop every day. The main server also serves as a SunRay server so I do most of my daily chores on a SunRay. I run Windows inside VirtualBox and I rarely ever turn on my windows PC anymore (the Windows instance in VBox also mounts from my main server). Inside my main server I have 2x 1TB drives, in a ZFS mirror setup, for the user directories and 2x400GB for the OS and scratch directories (all drives are SATA).
I'm very confident in this setup, also because I can yank out my drives in under 30 seconds in case of fire. The only thing I still have to do is put my backup server in a different room from the main server - that is a todo project for the near future.
Problem 1: If you are not home and your power supply decides to catch fire, you have lost everything.
Problem 2: If you are home, you better be spending those 30 seconds trying to get your butt out of the fire, not running after hard drives.
If you think your DR plan relies on yanking drives out, you're in serious trouble. One B&E or a fire and your data is gone. Now this may be perfectly acceptable to you. It is to a lot of small companies, until it happens to them.
Personally, I vault offsite on a daily basis as well.
Re: (Score:2)
I hear you, my biggest worry isn't fire itself, its fire after earthquake. In addition to my backup solution described above, I keep rotated drives with snapshots at work.
If my house burns down completely and all data is unretrievable I will have lost at most 6 months of data. Not all.
Re: (Score:2)
It won't be cheap to set up, but once it's going your only cost is going to be replacing drives that fail. It would be a lot cheaper than 500GB with Amazon's S3.
Re:Hard drives kept online (Score:5, Informative)
Re:Hard drives kept online (Score:5, Informative)
I tend to agree, however I'm a bit confused over what exactly is being requested.
"I've thought about buying a bunch of 4GB thumb drives....I have less than 500GBs and only save things important to me."
At first glance I thought you had 500gb you were trying to store, but then you mentioned "buying a bunch of 4GB thumbdrives" and I can't imagine someone buying 125 4gb thumbdrives to use for backup. So exactly how much data are you trying to store?
If less than 50gb, I'd suggest a few SD cards. 8gb SD is ~$11 [amazon.com], or 16gb for $30 [amazon.com]. While more expensive than hard drives per gb, SD cards are remarkably resilient, surviving a week in the ocean [letsgodigital.org], and a few in a ziplock bag stored in a safe deposit box would probably last close to forever.
SD will probably still be around at least for the next decade or longer. SD has already been around since 1999 [wikipedia.org] and all modern card readers read SD cards by default. SD slots are in nearly every form of consumer electronic device, and every manufacture of digital cameras uses SD except Sony and Olympus [wikipedia.org], almost guaranteeing the card readers will be around for many years to come.
I would suggest against USB anything since they're already discussing cutting the cord on USB and going wireless USB [intel.com]. While I don't predict that will happen overnight you wanted a solution that would be available decades from now, and wired USB might go the way of the parallel port [wikipedia.org], which was the standard external port in the 80s and 90s but was replaced by USB late 90s. Parallel port only had a lifespan of about 20 yrs and is no longer on modern PCs, and USB has been out just over 10 years [wikipedia.org] so it's feasible in 10 years PCs will no longer have USB ports, everything will be wireless USB.
Re: (Score:3, Insightful)
You can still get USB to parallel port adaptors.
The same thing will happen with whatever replaces USB.
I really hope we do not move to wireless USB. It will just be an extra set of security holes and other problems.
Re: (Score:3, Interesting)
[citation needed], please. Really, my understanding is that non powered hard drives are NOT good long term candidates due to 'stiction' (maybe not so much an issue now, maybe it is) and perhaps other problems.
(Warning: Anecdote time) I've had several previously good HDs in my junk box fail to start when I decided I wanted to play
Re: (Score:3, Interesting)
What's "striction"? I'm asking because I just switched-off my USB drive in order to save power. Maybe I should turn it back on & let it run continuously?
My suggestions:
- Home videos - nothing lasts as long as analog videotape. Twilight Zone episodes recorded in the 1960s are still viable today, whereas DVD-Rs have a nasty habit of self-erasing themselves in just 2-3 years. I think Super VHS tape is the safest way to store family memories.
- For files smaller than 20 megabytes, I upload them to yahoo
Re:Hard drives kept online (Score:5, Informative)
I'm not sure what "striction" is (and I suspect I don't really want to find out). "Stiction" on the other hand, refers (among other things, check out the Wikipedia link) to the tendency of the lubricants inside the HDD to increase in viscosity over time. After a long enough period of time, the motor can't rotate the platters. The fun thing about this problem is that the 'repair' involves brute force - banging the HDD to move things a bit.
Supposedly was a problem on older drives. My brief Google search did not yield any useful info. The problem with HDDs is that they are designed to be on and NOT designed for long term, powered off storage. That said, turning your USB drive on and off shouldn't be much of an issue. Remember the thing was probably made 6 months before it was installed in your housing and then sat in a warehouse for a while. The question is - can you expect a random HDD to fire up after sitting in storage for x years (x being larger than 2 or 3)?
What about SSD? (Score:3, Insightful)
If NAND flash SSD lifetimes are determined by write frequency, then wouldn't this be fantastic for archival storage? Just write the data once, then read it as many times as you like.
Re: (Score:3, Informative)
Really? Taken apart any modern hard drives lately? "Parking" is simply the act of moving the heads to an area of the platters not used for data storage. The heads are still very much in contact with the platters, but if the heads get bounced around, there's no data under them to damage.
Stiction is caused by centripetal force of the spinning platters gradually drawing spindle bearing lubricant across the surface of the platters. This happens gradually over a long period of time. Once that happens, if you lea
EASY! (Score:2, Funny)
Not enough history (Score:5, Insightful)
We don't have enough history on this tech to know what, if anything, will "last for decades". Possibly "paper" and "microfiche" might fit in that list, but those aren't the sort of things you're talking about. Best option I can think of right now would be to get a couple 500gig drives, put everything on both, and then put them in different areas. In 3-5 years, back them up to something newer, and repeat that every 3-5 years. Maybe in those intervening years, we'll have more data and newer tech that's demonstrably suited for what your needs are.
Re: (Score:2)
They'll want to be spun up every so often (6mo or so).
Re:Not enough history (Score:5, Insightful)
One drive should be "live" and the other archived. Considering we all own computers, throwing a 1tb drive into a box isnt so difficult. Hell, you could write a script to power it up once a month and then power it down, if people are worried about energy costs but dont want to keep it spinning 24/7. It doesnt need to be ever mounted.
Better yet both disks should be running in a RAID 1 array. This is a cheap solution, but its not a "toss in the closet and forget" solution. If this guy actually cares about his data I dont see why he cant spend 200 dollars or so for two drives and a raid 1 card.
I see this question at slashdot every couple of months. The answers are still the same. Keep it live on a disk until a better solution is found. Upgrade the disk every so often. That's it. Mods, stop posting the same damn question every month.
Re:Not enough history (Score:5, Interesting)
I let a lot of my early programs go. I had them on 9 track digital tape, at 6250 bpi. I had converted some of them from the lower bpi rates before. It used to be that each employer I worked for had one of those tape drives and I could access the files
when I wanted to. But not anymore. Besides, would I really want to convert FORTRAN programs?
In theory, I could have kept converting them to newer and newer storage media, but I didn't.
Later on, I had one of those QIC tape drives that could hook up to a floppy disk controller.
I think I have those files backed up on a hard disk somewhere -- I think. When I was going through old backups one year, I noticed I had the same directories appearing many many times.
I spent hours selecting the best version of everything and then backing up that too.
I haven't digitized all my vinyl record albums either, even though it is theoretically possible.
I have not scanned all the 35 mm slides or prints I took going back to the 1970s. At some point, the new stuff going on in your life gets more interesting than converting the stuff that used to get you excited 25 years ago.
Imagine how hard it is going to be to preserve people's papers after their dead. Do you want to keep converting someone else's stuff when you have a few terabytes of your own to attend to?
Re:Not enough history (Score:5, Informative)
The spindle bearings can seize up.
The problem isn't that the drive will inevitably die after 5 years, it's that it won't inevitably last longer.
Re: (Score:3, Informative)
Bit Rot [wikipedia.org]
Magnetic Tapes... (Score:2)
Even-though longevity of magnetic tapes has not been explored extensively, it is understood that if they are kept in a stable humidity and temperature environment with no light, that they should last for at least 20 years.
Re:Magnetic Tapes... (Score:5, Interesting)
Having a tape around for 20 years doesn't do you any good. 20 years ago, I was writing to 1600 and 6250bpi tapes. Today, my data center doesn't even have a drive that can physically read them.
Today's tape technology is no different. 3 years ago was writing to SDLT tapes. By next year, I won't even have an SDLT drive in my data center, having migrated everything over to LTO.
Yeah, I have round tapes in my offsite storage. I have 4mm and DAT tapes out there. We're just wasting money storing the media, since we have nothing that can read them.
If I could read the old media and extract a really old database, would today's database app be able to read it? Probably not. And could I install that app on today's OS? Probably not. And could I install the OS from many years ago on today's hardware? Probably not. Could I compile source from 20 years ago with today's compilers? In many cases, actually I can't. And if it really did all magically get compiled, is anybody around that can still knows how to run the app?
Don't forget that 20 years ago, many systems didn't have TCP/IP installed. In 1988, mine didn't - it was a combination of RS232-attached terminals and XNS-attached graphics workstations. Drive sizes were 80-160MB. A couple of MB of memory was a lot.
For those of you not still in school, ask around and see how many folks in your IT department can name the server that held your financial data 10 years ago.
Re:Magnetic Tapes... (Score:4, Informative)
So start building VM's of operating systems and software that are in use. Archive those. Far from perfect or complete, but it should narrow the scope of the problem a little bit.
As far as personal stuff, I think the best solution is to have 2 or more live copies of all important data and just migrate them to whatever makes the most sense at a given point of time, and then also have backups of stuff. That doesn't work with the question, but there isn't really a cheap answer to the question at this point.
Re: (Score:3, Interesting)
At my work (18,000 desktop Nat'l research lab), we have a special group to keep old hardware around, just so they can go and pull data from old archives.
Re:Magnetic Tapes... (Score:5, Insightful)
I think your post is very insightful, and I have an additional problem to throw into the mix: sorting through all the crap you've archived, even assuming you can read it all.
I don't know about you, but I've run lots of different backups on lots of different systems, and one of the problems that always comes up is just finding the revision of the file you want. People say, "I want the copy before I made this revision-- I think I did that about a month ago." Check the backups and there are no revisions from a month ago, but there are 20 from the month before. Next thing you know you're checking 20 copies by hand, and none of them are what you're looking for-- and that's even when your backup/archive system is working.
So when devising any kind of archive, I think it's at least worth considering, "How am I going to find what I'm looking for in 20 years?" Imagine yourself in 20 years, and you have every piece of data you've ever generated stored on some kind of media that holds hundreds of terabytes of data. You want to find some spreadsheet you made today (20 years ago). Maybe you don't remember exactly when you made the document-- you think about 15 years ago, but it's actually 20. You can't really remember what the filename was. You can't remember if you made it in Excel or OpenOffice, so you're not even sure what filetype you're looking for. What's going to be your method for finding that file?
I'm not suggesting it's an insoluble problem. It might be that it's not even a problem in 20 years because indexing/searching has become so good that your AI will be able to sort through terabytes in a couple seconds and make some good guesses about what you're looking for, but do you really want to rely on that happening?
Re: (Score:3, Interesting)
You can definitely do that today with DOS and Windows 3.1 on todays hardware. If we move away from x86 architecture, then everything is out the window, but for now, yes, we can install OS from many years ago on todays hardware.
Tried that lately? Not sure about DOS and Win3.1, but certainly Win95 had a boatload of timing issues on newer hardware.
Flash drives (Score:2)
Re:Flash drives (Score:4, Interesting)
By the way, I *have* had an SD card fail. It was in my digital camera the whole time, worked fine for a couple years, then quit. The camera itself showed no sign of damage, so I don't think it was abused. It was a Kingston, too, which I consider reputable.
Re: (Score:3, Informative)
Actually Flash/memory drives are sensitive to radiation. Long term storage without regularly accessing the drive can lead to situations where blocks go bad beyond the ECC/CRC capabilities of the drive to fix. If you intend to store valuable data on memory devices for the long term you should (a) use multiple redundant drives (b) use a file-system with block-level ECC/CRC error correction and redundancy (like ZFS) (c) write each block to the device twice in different location (i.e. an mirror on the drive).
Th
Re: (Score:3, Funny)
Ok, dudes, if he's got radiation problems, he's got bigger problems than data storage.
Not being answered (Score:5, Insightful)
Obviously this question hasn't been answered for the general public because this is like the 4th year in a row that this question has been asked on Slashdot.
Re:Not being answered (Score:5, Funny)
well the backups storing the questions have been lost.
Think Different (Score:4, Insightful)
I think the issue is that people are thinking about this incorrectly. You don't really want to 'archive' this data -- keep it with you! Keep it with all of the data that you are using day to day and back it up and move it along with that.
My home workstation still has files from 15 years ago on it. I've replaced the computer many times, had a few hard drives fail, etc. but I've always restored both current and 'archive' data from backups and kept going.
Re:Not being answered (Score:5, Insightful)
That's because there is no fully satisfactory answer. We'd all like a just do this, throw it in the corner and when you come back for it in 50 years it'll all be there sort of solution, but there is no such beast within the realm of affordability.
It's a problem with several aspects to it as well. Let's say there is a SATA drive out there that absolutely CAN sit in a safe deposit box for 50 years and then work perfectly every time. In 50 years, all computers will have whatever the successor to whatever replaces SaS and when you mention SATA, the old timers will all get nostalgic and go on about tying onions to their belts (which was the fashion at the time). You'll then have to take the decidedly NOT affordable step of having someone build you a one-off SATA controller that can interface with a computer of that time. That is, if you can get the old-timers to stop reminiscing about the Vista debacle of aught eight long enough to recall the specifications of SATA. Be sure to duck, some of them might throw a chair for ilustration.
Quality DVDs, archival storage, repeated backups. (Score:4, Informative)
Long term:
Use quality DVDs. Redo the backup on a schedule such that everything is re-backed up every three years or so. Every month, say, you make one DVD. Keep the backups in a climate controlled, dark, secure place, such as a safe deposit box at the bank.
Short term:
Back up everything you want to save to an external hard drive weekly. Every three months swap it with a drive kept in the safe deposit box.
Daily:
If you have a Mac, use Time Machine. If Linux, some sort of cron job running a Python script that copies /home to an external hard drive. If Windows, I dunno.
Re: (Score:2)
What I have in my home system is a small back-up drive sitting inside the PC. Every night, the drive spins up, personal/irreplacable data gets rsynced to it (therefore very little work) and is spun down.
Easy, cheap, and lets the HD work for the minimal time needed.
Also serves as a sort of recycle bin if I mistakenly delete something.
yes it works-for equipment failure (Score:3, Insightful)
right up until you have an enviroment diasaster.
(enviroment can be as small as a tiny fire in the power supply of that PC)
Theft of the PC? are you covered?
FFS it is IN THE SAME CASE!
Re: (Score:2)
This is really close to what I do. Running a home network, all the data is dumped to a RAID-1 and monthly copied to CD/DVD, which are kept in suitable storage space. This gives 3 levels of recovery:
1 - local hard drives have the data - manually done
2 - RAID on the network has a copy - scripted backups
3 - CD/DVD has a copy - manually initiated scripted backup
If I was truly worried, I'd make two CD/DVD copies and store one in Iron Mountain or something similar.
You can substitute USB drives for one of the CD/D
Re: (Score:2)
I can't imagine a guy with 500G of personal files. It must take a month just to read the titles of his various files.
But people are different. I'm not prone to believing in the media failure reports that we have all seen. For example I have a pile of floppy disks that are still intact after 15 plus years and I stored them like a barbarian. Hard drives also tend to last for me. And I suspect that any quality CD or DVD will last for quite a long time if handled
Re: (Score:3, Insightful)
Re: (Score:2)
It's not just porn and Slackware iso's.
Re:Quality DVDs, archival storage, repeated backup (Score:2)
Even quality DVD surfaces (on DVDs you can burn yourself) degrade quickly over a period of time (in my experience 2-4 years). Doing a re-backup every 3 years is too risky, it would have to be every two. In my case, with close to 1.6 TB of personal data (video, pictures, the works) it is not even practical, it would mean doing a re-backup of a DVD every two-three days.
Ask Slashdot AGAIN (Score:5, Informative)
As for the question itself, the answer is pretty simple, but unhelpful. Basically what it comes down to is that there is no safe place for your data. You're asking for the best type of basket to put all your eggs in. If you look at it that way, the solution is easier to arrive at. Your choices are A) spare no expense and build/buy the world's strongest basket and pray no flaw arises, or B) start copying your eggs around to all sorts of cheap baskets and continuously add more baskets in the expectation that the oldest baskets are going to fail.
Copy all your stuff to all your computers. Burn to DVD and/or CD ROM. Buy SD cards and USB flash drives. High capacity storage devices are so cheap now that you can keep all your valuable pictures of your vacation to Cleveland quite safe by constant duplication. That's the value of digital. Copies are perfect. Make lots.
Re:Ask Slashdot AGAIN (Score:5, Funny)
How many times has this question been asked on Slashdot?
It needs repeated backing up.
Not the media that's the problem (Score:5, Insightful)
Re:Not the media that's the problem (Score:5, Interesting)
There really is a simple way around this - and it is what I've done - I've got data 25 years old and it's still relatively easily manipulated with a little work. I've found floppy disks are relatively resilient, and old hard drives seem to keep their data for a long time. I've got a computer, display, keyboard, and associated peripherals stored for every generation of data that I kept:
1.I have a Commodore 64 with floppy drive and cassette drive stored in a box with the floppy disks and cassettes from that generation (late 70s/early 80s).
2.I have an IBM PC/XT with keyboard, a 5 1/4" floppy, 3 1/2" floppy, internal 20MB hard drive, and CGA monitor stored in a box with a load of 5 1/4" floppies filled with data from that generation (Mid 80s).
3.I have an IBM RS/6000 with display, keyboard, and mouse and internal 500MB hard drive loaded with all my docs and projects from that generation (early 90s).
4.I have a Pentium 2/300 PC * 15" monitor with windows 98, CD R/W drive, 3 1/2" floppy drive, and USB ports - and a crapload of CD's and 3 1/2" floppies full of stuff from that generation (Mid/late 90s).
When the current generation looks like it's going to be moving on, I'll put away a Core 2 Duo system with 1 TB of hard drive full of stuff with the different OS's I used loaded on it with boot manager (Ubuntu, XP, FreeBSD), a crapload of USB keys full of documents, along with burned DVDs etc. That'll take care of the "'00" generation.
The answer lies in not only archiving your data "of the generation" but the essential equipment needed to access it. I may have a heck of a time moving data off of my Commodore 64 - but I can at least see it and access it - I believe I stored a modem with it - so at worse I could set up a terminal server that it could dial into and dump data to. All the other systems I'm pretty sure I could recover stuff from - even if the PC/XT does have an MFM hard drive, etc.
Re: (Score:2, Insightful)
Wrong question (Score:4, Informative)
It's impossible for guarantee 100% storage integrity, just like it's impossible to guarantee 100% uptime. What you want to ask is what risk of data loss you are willing to take.
This page compares some of the options [sun.com] in terms of Mean Time To Data Loss (MTTDL). For the amount of space you're looking at (~500gb), a three-way mirror is probably sufficient to last for your lifetime.
But there's always the risk of fat-fingering "rm -rf" or having the building catch fire, so maybe you want to have two synchronized sets of mirrors, stored in different physical locations. Only you can decide if that's too paranoid for you (or not paranoid enough).
Simple solution (Score:2)
Paper tape - accept no substitute.
Apart from anything else, the standing waves you can get as it goes through the reader are alone enough to justify it.
Jesus saves, but Buddha makes incremental backups. (Score:2, Informative)
But seriously, I've had the same (but growing) data set in my /home for over 15 years, and going. I find the easiest way is to just keep it on my drive, and have a few frequently updated copies on external media (optical or solid or dirve) and to keep it on another PC too, disk space is so goddamn cheap. I also have a large music collection, and instead of wasting time backing it up onto optical media, I just keep it on both my notebook and PC, its unlikely both will fail at the same time, and incase of a
Gmail (Score:3, Interesting)
An archive is not a long-term backup (Score:5, Insightful)
There is absolutely nothing that you can put away for decades and expect to be useful. Your requirements are not simple - they'll actually very, very hard to meet, even if you want to throw a lot of money at the problem.
You don't know that a jpeg, for example, will be readable in 30 years. The format may be so deprecated that there might not even be a viewer available. Like my old Microsoft Works 4.0 documents - although I have the data, I have nothing that can read them unless I want to spin up an old Windows image, assuming that I can generate a virtualized environment that can support an old Windows (Windows XP probably won't even boot on any PC being produced 30 years from now). And some of that data is only a few years old, not decades old.
You should store not only the data, but also the applications that created the data. And the computer you need to run those applications. And backups of those. And then every few years, pull it all back and validate it and update as required.
You may have only 500GB now, but 10 years from now that will be 5TB. And then you need a way to actually be able to find something you added to your "archive".
I deal with this at work regularly. An archive is not a backup that you keep for a long time. It's much, much more than that. Once you start thinking about all of the issues that come up, you'll see that the media is the least of your problems.
Re: (Score:2)
Re:An archive is not a long-term backup (Score:5, Insightful)
I think we do know that a JPEG will be readable in 30 years. Formats that have been around for like 10-20 years like JPEG are going to be here for a long while longer; I'd say until the end of civilization at a minimum (and even then, it wouldn't be hard for people to figure out the format). The worse case is that in future generations only a librarian or data archaeologist would have the tool to open it. Given the open source nature of JPEG, more likely you'll just download a JPEG viewer.
MS Work 4.0 documents is completely different. There was always only one implementation, it wasn't open source, it wasn't a documented standard, and the life span of the format was small to tiny.
Re:An archive is not a long-term backup (Score:5, Insightful)
I think you're exaggerating the problem a bit. Formats like GIF, JPEG, and ODF will most likely be readable somehow in 30 years. They may not be the format of choice, but we have open source readers for those things, so for as long as lots of people have data in those formats, someone will be maintaining viewers that allow reading them and probably converting them to newer formats. Besides, it's not clear to me that we're going to come up with much better compression methods for static images, or that we really need to bother coming up with much better compression methods for static images, which means it isn't that unlikely we'll still be using JPEG in 30 years. I'm not saying it's a lock or anything, but it's not *that* unlikely.
Now, with a format like ODF, if adoption isn't bigger before something new comes along, you might have a hard time reading that just because of the relative obscurity of the format (which is a problem JPEG doesn't have). In that case, it will probably depend entirely whether enough people have enough valuable information in ODF that some developers somewhere think it's worth writing a viewer.
Yes, ideally emulation would be available for every obsolete platform, and we'd all keep VM images of all our old operating systems. We'd all keep all of our old applications to install on those images, and VM software would always be backwards-compatible meaning that we'd never lose anything. I'd love to know that someone somewhere is working on that, if only for historical preservation. However, for the individual who might have limited resources, it probably won't be necessary. If it ever becomes necessary for that to happen for most people, someone will be able to make a lot of money selling a solution.
In most cases I'd say the best bet is to stick to open formats, keep copies on multiple different media, and continually migrate to new media. So, for example, back everything up to a hard drive and create checksums for every file, and then burn multiple copies to DVD. In 3 years, pull them all out, check all the checksums for corruption, and copy known-good copies (and checksums)to your brand new 5TB hard drive, and burn a couple BluRay discs. In another 5 years, check the checksums again, get known-good copies, and copy them to your 50 TB SSD and burn a couple copies into your super-ultra-cool whatchamakallit.
Re: (Score:2)
The source for JPEG viewers is open. You don't need to run emulated code, there'll be native readers.
Sorry, it's insoluble. (Score:5, Interesting)
I believe this to be a serious problem with no good solution currently. That's the truth. You'll get lots of dismissive posts saying it's no big deal, but it is.
Forget media integrity. The problem is technology drift. Everyone thinks "ubiquitous" (as in every computer has a USB port) is the same as "eternal," and it isn't. Twenty years from now, your USB thumb drives and CD-R's may have their data physically intact, but only museums will have equipment that can read them.
It is a fantasy to suppose that you can successfully perform Sisyphus-like task of systematically recopying your data to new media and formats. The proof of this is the innumerable stories of big, well-funded organizations that have neglected to do this. If the NASAs of the world keep finding reels of tape with important data on it that can't be read due to technology skew, what makes you think that you can do much better?
(What makes me bitter is failure of vendors to give adequate warning when software updates remove the capabilities of reading file formats that were formerly supported. I once verified that my new Mac could read my old MFS diskettes, and did not notice when a software update to the OS removed that capability. Microsoft was less than forthcoming when they removed the built-in ability of Excel to read Multiplan files).
Re: (Score:2)
Scanners will still be around, in some form.
Paperdisk.
Then, file formats will be your only problem. Stick with open formats, and it'll be able to be figured out.
Re: (Score:2)
Twenty years from now, your USB thumb drives and CD-R's may have their data physically intact, but only museums will have equipment that can read them.
This type of statement always pops up in discussions about archiving, but I think it just means you haven't really thought it through.
20 years ago, computers were expensive and scarce.
Today, computers are cheap and plentiful.
What I'm trying to say is that the base amount of current tech is so vastly larger that, in 20 years, finding a guy who can pull an old piece of hardware from his basement will be much much easier than someone trying to do the same thing today.
Re: (Score:3, Insightful)
You raise some good points but perhaps protest too much.
Of course, it depends on exactly WHAT form your data is in. Text files will likely be readable until Kingdom Come. Microsoft Works files seemingly get deprecated every version. It is quite likely that common graphics files (JPEG especially, TIFF probably) will be readable for quite some time. JPEG especi
It doesn't exist. (Score:5, Insightful)
You are the analogy of an investor who wants a high-yield, low-risk, completely liquid instrument. The term is TANSTAAFL.
I maintain two (yes, two) USB external drives. Every couple of years, I migrate to a larger, or otherwise better medium. I use an incremental backup system (for me, cpio) that ends up keeping too much stuff, but at least I have the stuff I want if I need to get to it.
In a decade - in my case, four decades - one can accumulate a remarkable amount of crap, along with things one truly wants to save. I have a total of about 90 gig of actual data, plus a far larger amount of music and video, which I consider more or less disposable. It is not difficult, nor expensive, to purchase another external drive and copy the data. My oldest backup is on IBM 2314 disk pack, but the data still held on that disk is also present on my current backup, a WD 160G in a USB-1 enclosure. Sometime next year, I'll go to a 500 G drive in a USB-2 enclosure.
An important consideration is to periodically check to see that the data ostensibly held on a drive (or CD, or DVD) is actually readable. DVD/RW in particular has a tendency to get flakey over long periods of time, expecially if stored under adverse conditions (jammed in back of desk drawer, under sixteen pair of scissors, stapler, a box of pop-tarts, and four old coffee cups. I always keep my last few generations of backups, and if I find an unreadable datum, I make an effort to recover it from the previous backup.
While it may be stating the obvious, it's a Bad Idea (TM) to wait to back up data until you have a problem. I back up all of my data every week or two, and critical data, daily, without fail. Critical data is cached as a three-generation dataset (IBMese).
Good luck. There are no real solutions, just ways to cope.
Obviously the best storage method ever. (Score:5, Funny)
1) Encrypt your data
2) Tack it onto the end of some anime or porno mpg
3) upload to kazaa/gnutella/whatever
It'll still be circulating the net long after our grandchildren are dead.
This is actually a good idea (Score:5, Interesting)
This actually is a good idea. If the porn files were maybe one half sex imagery and one half encrypted private data, and there was no easy way to separate the two halves, then people would download, store, and upload the files in order to view the porn. Anyone who had data in the private section of the file could download it from various P2P sites.
The cost of filming and creating the porn file would be covered by the people who would be using the file for long-term distributed storage. Say a 1.5Gigabyte file that was an hour of MP4 video entertainment and 500 megabytes of distributed storage. The fees received by the producers for the storage would pay for the video production costs. Since porn is cheap to produce, this may solve the problem of piracy and secure storage at the same time.
Stone tablets (Score:5, Funny)
Again, Quality DVDs (Score:2, Informative)
Easy (Score:2)
Jungle Disk (Score:2)
Jungle disk frontend for s3. Easy, pay as you go and redundancy and multiple backups are taken care of by s3
Stop asking Slashdot about this... (Score:3, Informative)
... honestly, Slashdot - and others - have covered this time and time again. Nothing has changed. There still isn't a cheap digital storage medium that we know for sure -will- retain your data -and- be readable (in terms of media -and- the hardware to read that media) down to the very last bit for your grandchildren.
IF and when there's a breakthrough, I'm sure Slashdot is one of the first places you'll hear about it.. but it won't be in an answer to an Ask Slashdot.
http://it.slashdot.org/article.pl?sid=07/04/21/1257249 [slashdot.org] - Digital Media Archiving Challenges Hollywood .. and so forth and so on.
http://hardware.slashdot.org/article.pl?sid=06/11/20/2036247 [slashdot.org] - Archiving Digital Data an Unsolved Problem
http://hardware.slashdot.org/hardware/06/12/11/1714232.shtml [slashdot.org] - How To Choose Archival CD/DVD Media
http://hardware.slashdot.org/article.pl?sid=05/06/26/218250&from=rss [slashdot.org] - Archiving Digital History at the NARA
http://tech.slashdot.org/article.pl?sid=07/03/31/2141204 [slashdot.org] - How To Properly Archive Data On Disc Media
Yes, I realize that you stated "I'm not looking to start my own national archive; I have less than 500GBs and only save things important to me". However, it doesn't really matter whether you're archiving hollywood movies, NASA records or just your own random crap. If it is important to you - important enough that you want it to be "lasting for decades if possible" - then your concerns are the same as NASA's... and they're struggling with the exact same question.
The 'best' answer so far is one you will find in each and every single discussion on this - including this thread, so I'll just point you there:
http://ask.slashdot.org/comments.pl?sid=1061489&cid=26102825 [slashdot.org]
You mentioned 'cheap', as otherwise all the answers saying "duuuuude, ditch the digital - go analog!" might have some validity.. take a wild guess as to what it would cost to have thousands of photos transferred to negatives/prints, or video transferred to tape/film, etc. Plus you mentioned documents.. some of those may not transfer to e.g. paper (easily) at all depending on the 'documents' in question; e.g. CAD files.
Hard drives fail, but rarely at the same time... (Score:2)
Look, I know how you feel. I'm a professional writer and journalist, and the digital files I save are pretty much the only result of my life's work, so I try to save them for the future. I'm in the same position, but I seem to have a system that works.
Amazingly, (or sadly, depending on your opinion) I also have all of the documents and material I've ever created (starting with the disks from my Apple //e from high school in the mid 80's, to my emails and my first book's Word files from the early 90's, and e
Changing formats (Score:4, Insightful)
[From Slashwayback]: Dear Keypunch, I have data I want to keep for decades. Should I invest in a good card reader, or should I transfer my data to these far more efficient but newfangled "floppy disks"?
It's pretty ridiculous to expect one storage format to be viable for 'decades'. Not because it goes bad (even though it probably does), but because you're not likely to be able to maintain the necessary equipment for that long. If you find a storage solution, you need a retrieval solution to equal it. What equipment will you be able to find decades on that can access your storage, even if it stays good? You have no idea.
I've been maintaining a collection of Apple IIs and recopying the programs and data regularly (mostly through full HD backup, reformat with error block deletion, reformatting and replacing) to keep it readable. I have machines and data between 20 and 30 years old. I recognized long ago this had become a hobby in its own right, as most of what I had hasn't been of interest to me for many years. The little bits that have been useful have been transferred to newer machines and formats several times. That's decreased as more and more of it can be found easily on the web (previously FTP/gopher/etc.).
Get used to transferring your data to new formats as they come into widespread use, and recopying as necessary to keep them readable. Or else:
[From Slashwayforward] Dear Galactic EM Field Computing, I just found about 20 pounds of aluminized plastic disks that used to have data on them, but I can't read them to tell if I still want it. Is there any museum that might want these? Or are there still any operating plastic recycling centers that might give me a few bucks for them?
redundant storage on the web (Score:2)
I would suggest using at least two quality options from:
1. Apple's MobileMe (used to be called .Mac): you get only 20 gigs of storage with a basic subscription, but storage upgrades are cheap enough.
2. high quality managed (or partially managed) hosting - if you need this anyway for business use, get extra disk space. Make sure that their backups are regular and secure.
3. Other paid for storage options.
4. back up to DVD-Rs, and recopy every 2 or 3 years. I buy different brands, and rotate which I use. Redun
Two hard drives + fireproof safe. (Score:2)
whatever you do, do it twice (Score:2)
Assume that something will go wrong. So don't just keep one copy - make sure there are at least two. Keep them in geographically separate locations: maybe with a family member, if you can trust them.
Personally, I'd go for two different solutions: maybe one magnetic and one optical. However, whatever you decide on, make sure to get it all bac
There are no really good options (Score:2)
There just aren't any good long term options today for either consumers or businesses. Yes, CDs and DVDs can fail, but there's another problem - in twenty years are there going to be working devices that can reliably read that media? Maybe, maybe not. Earlier this year, the Storage Networking Industry Association [snia.org] (SNIA) started up a Long Term Data Retention group [snia.org] to address this very problem. Perhaps they'll be able to come up with something.
Until then the only "solution" is to migrate data from medium
Parchment Scrolls? (Score:2)
Clay tablets? Carving into marble?
"For decades if possible?"....... (Score:2)
Seriously, if you want something that can interface with a common personal computer, then what matters is not the storage medium so much as the device interface. Once upon a time everything had 5.25" floppies, printer ports and serial ports.
Just get a couple external USB2 hard drives, (two for redundancy) copy everything onto both of them, and there you go--ten years of good storage.
(unplug them from the computer and the wall power when you're not actual
Being that all the techology solutions can fail, (Score:2)
Yesterday I set out in search of a way to store my documents, videos, and pictures for a long time without worrying about them.
Try a different approach, why not simply *stop* worrying about them?
Let your own brain be your backup. Lost great photos and videos of relatives and friends? At least you will have some fond memories of them.
And when you die, and thus your backup is gone, then you can *really* stop worrying about them, like totally.
Friends + rsync over ssh (Score:3, Interesting)
I have gigs of photos (wedding, long lost family picnics, etc) and music that I can't bare to lose.
Find someone who doesn't want to lose anything either and setup rsync over ssh. Synchronize often, rsync is very friendly to bandwidth.
For now. (Score:3, Interesting)
There really isn't a good solution right now.
After thinking about it a while I realized that:
1) Most of the stuff on my computers could be replaced.
2) The one thing I would really, really hate to loose are family photo's.
3) Hard drives WILL fail sooner or later.
4) Tapes are reliable for a while but even in a climate controlled vault I have had tapes at work end up bad after a couple of years. (not to mention the pain in the neck it is to find a working legacy tape drive after 10 years)
5) DVD's will probably have the same issues.
My solution for now is redundancy.
Digital photos get offloaded to my Linux pc. I use a program called Digicam. .5T USB external disk drives with the "one button backup" program that is set to run nightly.
I have a bash script that syncs the new photos to a Windows share on my wifes pc.
My wife has one of those
When I have a couple of new directories of photos I run another script that compresses the whole directory and splits the output into a bunch of 45 megabyte rar archives.
I then upload them to Microsoft's free "Skydrive". Microsoft just upped their free disk storage to 25GB.
I also have some documents saved on the free AOL Xdrive.
I figure in a couple of years there will be a better long term storage option. It will probably be something like a solid state drive that lasts for two hundred years. At that point I will save everything to that and store it in my safety deposit box at my bank.
Re:Well (Score:5, Funny)
Remember paper? That actually roots after a period of time.
That's why paper is specifically prohibited on my network.
Re: (Score:2)
for 500 GB that would be a lot of CD-Rs. While CD-R lasts longer than DVD-R, i am still skeptical. Buying 3 500 GB hard drives every few years and using them in RAID might still be the best solution.
Re:Amazon S3 (Score:5, Informative)
Its not really that cheap, and not that simple to use for personal backups. Unless you are willing to write your own backup scripts, its going to be a headache.
Querying S3 for a list of stored files is *very* slow, and you only get 1k results per query. This means you have to index what files you put in S3 in a local db. This allows you to ask the db what files are there (and how to grab them).
If you only have a few files you can use the S3 browser extension for Firefox (or one of a many file system mounting, ftp simulating, etc tools). Just keep in mind the 1k file limit per query and box things in folders of no more than 1k items. Otherwise you will have a very slow browsing experience.
I have around 120 GB of family photos and purchased mp3s that I would like to store. To store 120 GB at .15 per gigabyte/month for 1 year would cost me: $216 (at $18 a month).
We use it where I work, with great success, but it would be much to much work for me for a personal backup system.
Considering the cost, I would go with a consumer targeted app (there are LOTS of them). A number of them charge a flat flee for "unlimited" storage. Beware of how you interface them. Some support windows only.
Re:Amazon S3 (Score:4, Informative)
Jungledisk [jungledisk.com] takes care of all the tedious backup stuff for you, and it is only a one time charge for the app.
But you're right, S3 isn't cheap. To store 500 GB of data would be about $75 a month, plus the $50 to put it on the server in the first place.
Re:Amazon S3 (Score:5, Informative)
That is an interesting suggestion, but 500GB would cost $900/year (plus transfer costs) which I don't call "dirt cheap". As far as being there forever... who knows if there will be an Amazon in 10 years? Amazon might be more stable than most hosting options, but forever is a long time.
Re:Amazon S3 (Score:4, Funny)
Re: (Score:2)
Well it's not dirt cheap for 500GB - that's $75/month. But who really has 500GB of critical data.
Many folk here probably make a living out of making sure company data stays around 'forever'. It can easily be a full time job. The lesson from that is that if you want data to stay around forever you need to look after it yourself or pay someone else.
S3 is a perfect solution for this. BeanBagKing needs to decide what data he really needs preserved forever then pay accordingly.
The other alternative is doing it y
Re:Amazon S3 (Score:4, Insightful)
The OP specified "decades if possible." So we have two problems here. One is that 30 years at $75/mo comes out to $27,000, which is a little pricey. For that amount of money he could probably hire someone to come to his house once a year, verify the readability of his media for him, and transfer them to new media as the old ones become obsolete. $1,000 for a few hours' work? I'd take it.
The other problem is that the probability that Amazon S3 will exist in 30 years is very low. This is basically the problem with any possible answer to his question. There isn't any computer-related service or equipment that you can be sure will still be there in 30 years. A more realistic goal would be to do it in 10-year steps; if that's all he wants, then the shoebox full of flash drives should work fine, and then 10 years from now he can transfer those data to something else.
I initially misread that as 500 Mb, which is about the amount of critical data I have that needs backing up. 500 gigabytes is kind of a crazy amount of data. One way to get that much might be that he has a gigantic collection of mp3s, or possibly a moderately huge collection of music in a less lossy format. But then that's not critical personal data, it's just a music collection. And the chances are that as the decades go by, he'll realize that the music he thought was so important and wonderful in 2008 no longer seems so important to him. I know plenty of people who still have their Kool and the Gang LP's from the 1970's, but it's not like they're willing to spend a thousand dollars a year to obsessively maintain them.
Re: (Score:3, Insightful)
> I would figure that it's video... specifically, probably family home movies.
Digital video has opened a HUGE new can of worms. We have problems even today viewing video created just yesterday (especially over the Web) because of all the myriad codec standards.
Imagine what it'll be like in 20 years-- anything other than NTSC, PAL, or SECAM will be effectively extinct.
Re:Amazon S3 (Score:5, Insightful)
A lot of people shoot home movies and then become obsessed with preserving the footage. That adds up fast.
It seems that deciding what isn't important is a hard part of backing up.
After a loved one dies, even the lowest quality outtake with a thumb covering half the shot can be a priceless memory.
Re:Teganography (Score:4, Funny)
Actually, its steganography, but I used typoganography to hide the "s" :-)
That's my story, and I'm sticking to it ...
Re:Amazon S3 - until? (Score:3, Informative)
Yeah - sure - until Amazon goes out of business or gets bought and then the new owner dumps the service and you're S.O.L.
RS
Not Amazon S3 (Score:5, Informative)
Re: (Score:3, Insightful)
If your data is not in your possession, how do you know others won't see it or edit it without your permission?
Encryption ?
Re:Not Amazon S3 (Score:4, Insightful)
Re:Not Amazon S3 (Score:4, Insightful)
If someone's willing to brute-force a password to change your data, they're willing to copy your DVD's ISO, edit it, and burn a new copy to replace it with.
Re: (Score:3)
yes..
But, they have to climb the wall, get past the guard dogs, fool the electronic entry system, climb through the ductwork, fool the motion sensors, crack the safe, then manage to get off the premises alive before the response team gets there, and even THEN they will have problems defeating the ROT13 encryption my backed up savegames are protected with.
I take PBEM games Very Seriously.
Re: (Score:3, Insightful)
It actually kind of does.
In particular, modern RSA, even considering future advances, would require fundamentally different technology (quantum computing) to make it feasible.
By "feasible" I mean that if quantum computing doesn't become practical, a 4096-bit RSA key will survive the heat-death of the universe.
Crypto is very, very rarely the weak point. It's almost always how that crypto is used.
And by the way, mods -- blanket statements like the parents' are easy to come up with, and easy to make sound inte
Re: (Score:3, Interesting)
I have trouble with this in my mind, since so much of my work is devoted to making sure that information always exists and is accessible all the time. I look at these personal solutions for backup, and I'm so used to evaluating enterprise-type products that I scoff.
I guess that I don't know anymore what is appropriate for 'home users' when they say they want to keep data long-term. The submitter stated that tape drives were inadequate when that's still the most reliable method that enterprises use.
Sure, the
Re: (Score:2)
Cheers
Re: (Score:2)
I've just put a Drobo into use as well. My usage is two tiered:
1. Time Machine Backup for all the machines in our house
2. Archival of documents/media/stuff that we don't need immediately but may someday want
In order to make #1 to happen I connected Drobo to my existing Airport Base Station. Time Machine is happy, and functioning on all machines.
The cool factor with Drobo is that it gives you a lot of convenience of a RAID system without the administration issues. It allows for drive failure, it also su