How Do You Backup 20TB of Data? 983
Sean0michael writes "Recently I had a friend lose their entire electronic collection of music and movies by erasing a RAID array on their home server. He had 20TB of data on his rack at home that had survived a dozen hard drive failures over the years. But he didn't have a good way to backup that much data, so he never took one. Now he wishes he had.
Asking around among our tech-savvy friends though, no one has a good answer to the question, 'how would you backup 20TB of data?'. It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.
So I ask fellow slashdotters: for a home user, how do you backup 20TB of Data?" Even Amazon Glacier is pretty pricey for that much data.
Asking around among our tech-savvy friends though, no one has a good answer to the question, 'how would you backup 20TB of data?'. It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.
So I ask fellow slashdotters: for a home user, how do you backup 20TB of Data?" Even Amazon Glacier is pretty pricey for that much data.
Hmmm... (Score:5, Funny)
I would say use floppies, but I'm kind of old and out of touch now.
Re: (Score:2, Funny)
I would say use floppies, but I'm kind of old and out of touch now.
5 1/4" or 3 1/2"?
Re:Hmmm... (Score:5, Funny)
5 1/4" or 3 1/2"?
8". How young are you?
Re:Hmmm... (Score:4, Informative)
Re:Hmmm... (Score:5, Informative)
At the very end of the 5 1/4" floppy era, the "High-Density" floppy used the same data rate, tracking, and recording density as the 8" 1.2M floppies. They were, in fact, 1.2M 5 1/4" floppies. Which is why their formatted capacity was different from 3.5" "high-density" equivalent, 1.44M.
Other than electrical needs (as 8" floppies often had their spindle motors directly powered by 120VAC line current), the high-density 5 1/4"s were used as a drop-in replacement for 8" floppies in the hobbyist retrocomputing community. (Not collectors, though; they'd want to keep the gear as cherry as possible.)
Re:Hmmm... (Score:4, Funny)
Re:Hmmm... (Score:5, Insightful)
Which, if it takes about 1 minute to load each one, it will take you a mere 243 years to do the backup.
Re:Hmmm... (Score:4, Informative)
20TB = 1.33LoC
Re:math majors (Score:5, Funny)
Now this joke has really come full circle.
Re:Hmmm... (Score:4, Funny)
Neither, 8" floppies would be the way to go.
Hard sectored or soft sectored?
It would be best to decide up front before putting in the order for 80 million disks.
Re:Hmmm... (Score:5, Informative)
Punch the hole and you can flip them over to double your capacity.
Re: (Score:3)
This was the greatest trick ever.
Jacquard loom punch cards (Score:4)
I'm backing up my 40TB music library on Jacquard loom punch cards.
Added bonus: You can use the punched cards to make fabric. ...as a sweater!
Right now I'm wearing Justin Bieber's "Love Me"
https://web.duke.edu/isis/gess... [duke.edu]
Re:Hmmm... (Score:4, Funny)
Re:Hmmm... (Score:5, Funny)
Re:Hmmm... (Score:5, Funny)
Punched paper tape has better longevity than either floppies or optical media.
If you're going for *actual* longevity, you can't beat fired clay tablets. (Yeah, I know they weren't fired originally, but you have to decide how much you value your MP3 files. I'd certainly take the extra time!)
Re:Hmmm... (Score:5, Funny)
Boxcars / Gigabyte (Score:5, Funny)
Re:Boxcars / Gigabyte (Score:4, Funny)
Well sure, latency is a bitch but imagine the throughput once it got moving!
Re:Hmmm... (Score:5, Informative)
At 10 characters per second, the backup would take 63,419 years(*) and require 659 TJ or 0.2 TWh of power to complete. I have a customer that still uses paper tape. It lasts and lasts, and I have only replaced the reader once. The punch needs a new power supply every 20 years or so.
However, 63,419 years is a long time to wait for a backup to complete.
(*) this assumes that 1 TB = 1,000,000,000,000 bytes. It takes almost 70,000 years if you add the extra 10%.
Re:Hmmm... (Score:4, Informative)
Re:Hmmm... (Score:5, Funny)
Seed a torrent of it as an encrypted file named "porn.zip" or similar. You'll have it backed up on the cloud for free in no time and available for all of eternity.
Re: Hmmm... (Score:5, Insightful)
Well, it's local storage to a computer somewhere. If that computer can run Backblaze, then super.
Re:Hmmm... (Score:5, Insightful)
Of course, you might want to make more than one copy of your data. https://www.amazon.co.uk/dp/B006K1FSKA [amazon.co.uk]
Re:Hmmm... (Score:4, Interesting)
The drives, however, are not cheap. New drives appear to start at around $1200. Used drives are all over the place -- I've seen some on eBay with an opening bid as low as $350. Also, all LTO drives appear to have either an LVD SCSI or a SAS interface, which means you'll also need a controller card. There appears to be no such thing as a SATA LTO drive.
Plus you get to re-live all the joys of selecting tape vendors, and placing bets on whose tapes are going to last for 20 years.
Re:Hmmm... (Score:4, Informative)
Tape drives need the full SCSI command set, not the trimmed version that made it in to SATA (I'm not sure there's even a "(01) REWIND" supported in SATA).
LTO tapes stored reasonably ("keep in in a cool dry place", as the song goes) should last 15 years from any vendor, as that's in the spec, and there aren't really bottom-feeder vendors for LTO.
reduce the amount (Score:4, Interesting)
At home, I didn't feel like paying for 2 large arrays to store my data, so if I rip any media, I always rip it to DIVX. 800 MB for a DVD or even bluray rip is a great economy, saves me money on primary storage and also enables me to back it up. I accept the loss of quality as I can always reference the original media if I want.
Another option in the future may be subscription services which have HD content, thus eliminating my need to roll my own. We'll see what happens there.
Re:reduce the amount (Score:4, Informative)
20TB is not out of the world. With a RAID of 4TB disks you can cover that at home, and it doesn't need to be on all the time. Maybe you can reduce the amount of disk usage by reducing duplicate content using bup [github.com] or an appropriate FS.
Re: (Score:3)
20TB is not out of the world. With a RAID of 4TB disks you can cover that at home, and it doesn't need to be on all the time.
Sure, it's easy to have 20TB of usable disk space (I've got forty 2TB drives spread among 5 servers at my house), but 20TB of "must be backed up because that's the only copy" is a little unbelievable for a home user.
For example, I have 700 Blu-Ray movies that have been ripped and re-encoded to take about 2TB of disk space. If I had 30-40TB available, I might store the raw Blu-Ray images, but then I don't need backup, as the data is easy to re-create. So, I'm a little skeptical that the "friend" in TFS had
Re:reduce the amount (Score:4, Interesting)
I use Greyhole [greyhole.net] for media and document storage. It handles disks of unequal size (currently running one 3TB and two 1.5TB drives), and you can choose the level of redundancy you need. In my case, movies, TV shows, etc. get a single copy (one file exists on one drive), while documents and photos get two copies (one file exists on two drives). If a drive goes bad, you only lose the files on that drive...and only for the files for which you selected no redundancy. With redundancy, extra file copies are recreated on the remaining drives from the surviving copies; this process is most likely less stressful on the disk set than a RAID rebuild.
My movies, TV shows, and music are backed up to BD-R, stored in a binder at work. They hold ~20GB each, as I'm using dvdisaster [dvdisaster.net] to guard against media errors. When a 2TB drive failed, I brought the backup (currently about 190 discs) home and restored the files that had gone missing. Backup and restore are managed by scripts, with information about what files are on what discs held in a MySQL database that gets periodically backed up off-site as well. The initial backup took several months (on and off) to finish, and the last time I needed to restore, it took about a week, but now I just burn a disc when I have about enough new data to fill one. Burning and verifying takes a few hours, but it's something you can start and walk away.
Re:reduce the amount (Score:5, Informative)
The problem with RAID-5 is that you are 2 disks away from failure and rebuilds often kill the disks.
No. The problem with RAID-5 is that during a rebuild, there is a reasonably possible chance you could have a UBE, and lose one bit, making perfect recovery of the array impossible. Only a stupid controller would consider a UBE to be a failed drive and trash the entire array. On RAID-6, you still have the same possibility of a UBE, but the chances that two separate drives would experience one on the same exact block during a rebuild are so astronomically slim as to be irrelevant.
Re: (Score:3)
Re:reduce the amount (Score:5, Funny)
I do the exact same thing with high res pictures. I immediately will take the full resolution raw image and convert it down to a 320px gif. Or maybe a 10% quality jpeg. You get great economy that way too. Who wants to keep a 30+MB image around when you can have almost the same thing in 10kB instead!
Re:reduce the amount (Score:5, Insightful)
I wasn't looking for it to go anywhere really other than pointing out the absurdity of saying that taking a bluray rip down to a 800MB divx rip results in just an acceptable loss of quality.
I'm by no means a audio/videophile snob, but you either have a blind and/or deaf if you can't see a MAJOR quality deficiency with a 800 bluray rip. What's the point of having a bluray movie if the first thing you normally do is make it look like crap?
Re: (Score:3)
To be fair, 800MB is still better than your standard DVD rip, which is probably 200-400MB. But yeah, 800MB is still lame; 2GB would probably be a much better copy. It'd also help if he dumped his crappy DivX;-) codec and moved to something more modern.
Re: (Score:3, Insightful)
> I always rip it to DIVX. 800 MB for a DVD or even bluray rip is a great economy
I do that as well, but I found out to my horror that all my DVD's had become unreadable over time. So, probably good idea to test your backups from time to time
Re: (Score:3)
That's why you should reserve 10-15% of the disk for parity data. While DVD-R format has built in error-recovery at the sector level, by the time you figure out that the disk is going bad it is too late. By adding even more recovery data at the file level, you can treat the disk errors as an early-warning system, then use the recovery data files
Re:reduce the amount (Score:4, Informative)
Re: (Score:3)
x264 is great, but why mkv? I've had terrible compatibility with mkv files in portable players or tablets, and proprietary software. Many won't even recognize them. I rarely have issues with mp4. Sometimes a user has no choice on software/hardware.
Re:reduce the amount (Score:4, Insightful)
Agreed.
Regardless of whether or not 20TB is hording / excessive / inefficient, what it almost certainly is is replaceable. Let's face it, you aren't CERN, most of you data is probably media that you can reacquire with relative ease. It's not being stored because it's irreplaceable it's being stored because it's convenient. A RAID isn't too bad, but add in managing backups and where has that convenience gone? If it costs $10+/month to backup your ripped/downloaded movies, why not just sign up for Netflix?
Just make a list of all the replaceable data (e.g. videos you have the original disc for) you have and then buy an external hard disk / Blurays to back up the rest. If you lose your RAID, well, it'll be annoying to rebuild, but you built it once... (Besides, I doubt you could restore 20TB over residential internet less time!)
Re: (Score:3)
Agreed.
Regardless of whether or not 20TB is hording / excessive / inefficient, what it almost certainly is is replaceable. Let's face it, you aren't CERN, most of you data is probably media that you can reacquire with relative ease. It's not being stored because it's irreplaceable it's being stored because it's convenient. A RAID isn't too bad, but add in managing backups and where has that convenience gone? If it costs $10+/month to backup your ripped/downloaded movies, why not just sign up for Netflix?
Just make a list of all the replaceable data (e.g. videos you have the original disc for) you have and then buy an external hard disk / Blurays to back up the rest. If you lose your RAID, well, it'll be annoying to rebuild, but you built it once... (Besides, I doubt you could restore 20TB over residential internet less time!)
Some people have different use cases. A few years back I was visiting a friend in the boonies in Egypt and brought a TB of American movies and music along explicitly for her (she was putting me up for free while I was on a research project). With my 50Mbps connection and 250GB monthly cap, I could recreate the entire shebang in 4-5 months, but with her iffy ISP, she couldn't hope to download everything from me in her government's lifetime.
Re: (Score:3)
Regardless of whether or not 20TB is hording / excessive / inefficient, what it almost certainly is is replaceable. Let's face it, you aren't CERN, most of you data is probably media that you can reacquire with relative ease. It's not being stored because it's irreplaceable it's being stored because it's convenient.
There's a large, flawed, assumption running through this thread that it's "easy" to reacquire all media.
Between torrents and usenet, trying to find good rips of content that's over 6-12 months old is often impossible. Good luck trying to find any older show that's not sci-fi or super popular. Often studios don't sell DVDs or Blurays of these shows anymore, if they even did to start with.
Re: (Score:3, Insightful)
If you've already got one large array then you are already by definition half way there. If you then decide not to go the rest of the way then you are at the same time being both extravagant and a cheap bastard. It's a wonderfully stupid paradox.
If you've got one then you should get the 2nd one or not bother with the first one to begin with.
Re: (Score:3)
Oh yeah, 20TB over VPN is *very* useful. /snicker,snicker
Crashplan (Score:5, Informative)
Crashplan has unlimited storage. I use their home plan; it's unlimited for up to 10 machines. I think I am backing up about 6TB there now.
I agree but... (Score:5, Informative)
Re: Crashplan (Score:5, Interesting)
Crashplan offers unlimited storage, yes, but they limit it indirectly by slowing down uploads.
I recently paid for a crashplan account to back up ~6TB of media, and at the speeds I'm seeing the initial backup is going to take more than a year. I have 100Mbit/s fiber at my home and can max it easily with other services.
So for 20TB, it's going to take many years to back up. I don't think that's a practical backup solution. There's a decent chance you're going to lose your data before the initial backup completes. And if crashplan goes under, you have to start all over again with the next "unlimited except for rate" provider, and have no backup in the meantime.
Re: (Score:3)
While I've never had the throttling issue that you have, I do want to point out that they accept seed drives if your initial backup is taking a long time.
Re: (Score:3)
Re: (Score:3)
Re: (Score:3)
Re: (Score:3)
Hard drives + Robocopy (Score:5, Informative)
External hard drives in USB cases + Robocopy works great for me.
Re: (Score:3)
Well, you might not need to have twice as many. It just depend how much you are willing to pay.
He is using a RAID-array as his primary storage. You will only need the other copy as a disaster recovery. In case you accidentally delete or format or the hardware RAID controller fails and you can't replace it anymore. Or something else stupid like that.
So if you have a RAID5 of your originals and a copy on non-RAID disks and do regular checksums of your data to compare if reading the data from the RAID and non-
You can just plug in an external drive, (Score:2)
If you want to hoard bits... (Score:4, Insightful)
> It's not like you could just plug in an external drive [...]
Why not? Maybe not one, but 10 or 20 of them.
Re: (Score:3)
An actually contemporary tape drive(and a machine capable of keeping it fed when it is running full bore) is Not Cheap; but the fleaba
Re: (Score:3)
An actually contemporary tape drive(and a machine capable of keeping it fed when it is running full bore) is Not Cheap; but the fleabay shit that is cheap tends to offer painfully mediocre capacity and unknown reliability. Disks, by contrast, have a cost of entry that basically starts at zero and scales more or less linearly with the number of disks
This is absolutely the best statement of this ever. Everyone who claims that tape is the One True Backup doesn't factor in the startup cost of $2-4K for a tape drive that can handle reasonably large capacity tapes, the hardware to connect it to a computer (many of these tape drives have fiber-channel as the only option), and then the cost of some kind of changer if you want any amount of automation.
For home use, hard drives are by far the cheapest and most convenient method, as long as you are in the less
Re:If you want to hoard bits... (Score:5, Informative)
The dataset isn't that huge. Tape can write at speed at least as fast as disk - LTO-5 writes at up to 280MB/sec - far faster than you can read the source at which isn't likely to be fast disk. The seek for a single-file restore will be slower than disk but after the initial seek, the read will be as fast as from a typical archive disk (no, you're not archiving 20TB to SSD, nor are you storing the source data on SSD either)
However, the change rate for this application is likely to be low. That makes it very feasible to do random testing from the new backups where a minute to do the tape mount/seek is not a problem. You won't be writing more than a single tape in any single run (LTO-5 is ~1.5 TB of uncompressed data).
For $2K, you'll have the LTO-5 drive. Add $500 for 20 tapes and you can back up the entire set (once) plus a bunch of incrementals. I haven't done the math with LTO-6 which is faster and holds more data. If you want multiple generations, tape is a lot cheaper per TB than disk. The initial drive cost hurts but after that, the price is good at $15/TB or so.
Similar Situation (Score:2)
Re: (Score:3)
You can get an LTO4 SAS drive for ~$50 on ebay, they do 800GB native per tape, so typically ~1.2TB per tape for mixed content (obviously if it's all compressed media it will be much closer to native). 10-20 tapes doesn't seem that bad (we send that many offsite daily). The tapes will cost you ~$20 each unless you're willing to go used (ewww).
Backblaze (Score:2)
Ah, "unlimited"... right. (*cough*) (Score:5, Informative)
These "unlimited" claims always turn out to be lies. When will we learn?
My friend paid for an "unlimited" account from JustCloud for backup. He stored 1.8 TB on it and then they "fair use"'d his ass and canceled his account. They didn't even give him a refund for the rest of the money he prepaid.
Re: (Score:3)
Fraudulent Capitalist Practice Spun into Argument Against Communism. News at 11.
github (Score:2)
Do something about your hoarding problem (Score:4, Insightful)
"My friend (read I) lost 20TB of pirated content! What should my friend have done different?"
How about, ask yourself, how much of that content were you intending to ever consume again. Yeah, you can most likely delete 95% of it, that's 1TB of content that you might use again.
Hoarders! *lol*
Re:Do something about your hoarding problem (Score:4, Insightful)
Not all of us have access to the time machine required to know *which* 1TB that is.
Are you willing to share yours?
Re:Do something about your hoarding problem (Score:5, Insightful)
This problem isnt unique; most people have trouble curating their data. That doesnt change the fact that the problem is mostly self-created, and the best solution isnt to find another place to stuff the 20TB. Its to take the time to cull it down to a reasonable size and then back it up.
Re: (Score:3)
You don't have to cull it down, you just need to organize it into logically distinct groups and assign them priorities. Hoarding isn't the problem, the problem is assigning too high a priority to the hoarded pr0n as compared to the really important stuff.
Contents: Documents, source control repositories, user preferences, email archives
Maximum Size: 10GB
Protection: 3-way Mirror + Snapshots + Offsite
Total Space Required (way upper bound): 150GB
Total Cost: $3 a month for Crashplan
Contents: Pe
Re: (Score:3)
It most definately IS worth curating your inbox. One of the toughest issues ive seen WRT backups and email performance is absurdly large PST files. Guess what: that PST file is marked as "modified" every time you open outlook, and most online backup systems will mark it as needing to be re-backed up each time leading to some absurd bandwidth usage. Theres also all of the issues with potential for a single file to get corrupted, difficulty in repairing it (ever tried to scanpst a 25GB pst?), performance
Re: (Score:3)
This is why I archive my email into separate bins every 60mb or so (which means about twice a year) -- anything larger becomes unwieldy. It's a nuisance, but not nearly the nuisance of trying to search the file with an external tool because it got munged and won't open in the mail client.
[This is with SeaMonkey, which still keeps mail as plaintext, thank ghod, so any good text editor can root about in the file at need.]
Re: (Score:3)
Good luck. (Score:4, Informative)
A quick check at one service which lists such large amounts, you would be looking at almost $20k/year to keep a single offsite copy of that. That is the posted price however, I imagine that is enough that you could shop around and find a deal, but, a deal is still going to be prohibitive for most people.
At 20 TB I would start thinking about one of two things: Tape, and/or git-annex.
Unless prices have changed since I last looked and the scales tipped, tape has the advantage of being cheap. Of course, you will need to test your tapes occasionally and likely want 2 copies just in case, but, at that point you are invested in tape, may as well.
The other possibility is git-annex and lots of drives, but you can mix types. That way you can keep a catalog of your library and information on where it all is, and how many copies of each thing you have.
Of course, any way you slice it, each physical piece of media is something that can fail so you have to occasionally test to ensure redundancy.
Don't hoard (Score:5, Insightful)
He could have always bought a sufficiently large tape-library from ebay - but I guess the data wasn't worth that much.
That's always the first pair of questions to ask: how much is it worth and how much would it cost to recreate?
If the answer is somewhere between "I don't know" and "Well, it's not that much", then he just should stop hoarding that much stuff.
He could have built a filer with ZFS and sent daily snapshots to a 2nd filer - but that wouldn't have helped him if the house burnt down...
My solution (Score:5, Funny)
Figure out the theory of everything.
Then you can always recompute your data from scratch.
Rsync and Bluray and maybe dedup (Score:3)
Why back it up at all... ask the NSA for a restore (Score:5, Funny)
one bit at a time (Score:3)
Plan for backup before you buy (Score:5, Insightful)
Whenever you buy storage, you should buy the necessary backup capacity at the same time. You should never buy storage without buying backup capacity. Budget for it right from the start. If you can't afford the backup, you can't afford the storage. This may mean getting half as much storage as you'd like, but that's just the way it has to be. You probably wouldn't buy a car without an engine. It wouldn't do its job. So don't buy storage without backup. If you do, you have a storage system that can't do its job.
Hilarious (Score:5, Insightful)
It's not like you could just plug in an external drive, and using any cloud service would be terribly expensive. Blu-Ray discs can hold a lot of data, but that's a lot of time (and money) spent burning discs that you likely will never need. Tape drives are another possibility, but are they right for this kind of problem? I don' t know. There might be something else out there, but I still have no feasible solution.
Lets start from the top: You *can* plug in an external drive, it's called a complete hardware duplicate of your array (or perhaps for space/cost consideration, a single disk based copy held offline and synced regularly). Not hard and not terribly expensive (i would go with this solution personally). Cloud? Yep the bandwidth and storage even on something like Amazon Glacier would be prohibitive to all but the most financially independent geeks. Bluray doesnt hold enough (even at 50gb/disc you need 400 of them, groan). So, tapes? You bet your ass tapes are designed to do exactly this task, why do you think they are still in use? You can get individual tapes at 1/1.5TB, but for a one man operation they are probably going to cost you more than the first solution (offline spinning disks) and they are a pain to manage properly.
Now what is this doing on ask slashdot? A pencil, some scratch paper, and 15 minutes between amazon.com and newegg.com would tell you the prices of every solution. Oh, right, they need a chance to tee up some targeted ads for Carbonite, Mozy, Crashplan, etc.
Backup only the best one (Score:3)
How about backing up only the crown jewels of the collection?
Make a directory like /entertainment/premium and put the best stuff there, with a 4 TB limit. Rotate two external 4 TB HDDs and copy the stuff over periodically. Put a little sticker or some other mark on the newest, so you remember which one it is. If your main RAID array fails, build a new one, and restore the premium stuff from the most recent one of the two external disks.
Re: (Score:3)
Why would you only back up the nut shots from his porn collection?
Amazon Glacier (Score:3, Interesting)
I use Glacier and its great. 20 TB is about $200 a month which to me does not seem like all that much money for backing up that much data. The biggest problem from a home users perspective is getting all of that data to Amazon. Hopefully he lives somewhere where fiber is available to his house.
Connect a raspberry pi and (Score:5, Funny)
Connect a raspberry pi and configure it as a backup server and let it copy all to /dev/null... ...
Then put aside the money you would have invested in a "better" solution, put it in a safe bank (under your mattress)
and wait until you need to restore something..
Most probably you'll enjoy the money more
No 2 ways about it (Score:3)
1- if you need to backup 20 TB today, you need to budget for 40TB in the medium term.
2- a backup is off-line, off-site, tested, and multiple. The "multiple" part is pricey, and the other 3 you can get cheapest with a PC filled with HDs. Or two (I'm making do with one). $200 for the BC, $150 per 4TB HD x 5 = $950. Hide that backup in a place safe from theft, floods, fire...
TAPE (Score:3)
There are many > 1TB tape back up systems, many with very high speeds, assuming you can feed it data fast enough.
I have to wonder though.. 20TB for a single person? I'm not gonna do the math but that sounds like so much stuff to be impossible to listen/watch all of it.
But at least he has proven once again, RAID is not a backup. RAID will merrily do what ever you wish, including copying drive corruption.
Comment removed (Score:5, Funny)
Stop being cheap or collecting so much crap data! (Score:3)
What does it mean that he didn't have "a good way to backup that much data, so he never took one"?
The concepts behind backing up data have not changed. You need to manage the size of your data to redundantly fit into the storage of your system. So either pony up the cash and time to properly store your files, stop collecting TBs of crap, or stop complaining about losing it when your system crashes.
It's frustrating to see people continuously complaining about how they have too much data to back up cheaply and conveniently. It's even more frustrating to see them complaining about losing all of their data because they didn't back it up properly.
I think that the main issue is that most people do not realistically or conservatively plan their actual storage capability. For example, it seems like 90% computer users believe that having 4 TB of hard drive space means that they can safely store 4 TB of data.
After a conversation about scratch space, redundant drives, and timestamped backups, they then will grudgingly agree to allocate 25% of their available storage to RAID/Backup space, which obviously does not get the job done! Very few are willing to accept using 66% of their available hard drive space for RAID and Backups, which is really the minimum metric for any sort of storage longevity.
20 TB is an awkward amount of data for a non-corporate individual to be storing. It's more data than most people actually need for their media and it is getting into a very expensive price range to backup for basic music/movie content. (By expensive, I mean that it would be cheaper to just re-purchase the media rather than back it up.)
Please post Tape backup ref (Score:5, Insightful)
To /.ers saying that 1TB+ tapes would be a good idea to do this backup, please:
Add some references and price of such hardware and media that would suit best home usage.
The Anti-Solution (Score:3)
I'm just speaking generally here, there are certainly cases where someone would need to back up this much data, but for your home media library? If we're talking movies, 20 TB is roughly 20,000 movies (for sake of argument, I'm not considering music). At what point is this just digital hoarding? I used to keep a large collection of movies, mostly pirated, and eventually realized that:
a) I was spending more time and money managing the collection then I wanted to. b) That I rarely watched many of the items in my library. c) That I was placing myself in legal jeopardy by storing so many illegal copies. d) Anything I did want to re-watch I could get from Netflix, the public library, or download.
Music would be slightly different, as I could see where music is in some kind of constant rotation, but again, how much of it are you actively using? I'm just playing devil's advocate here, but I think this kind of collecting/hoarding is a byproduct of pre-internet scarcity.
Plenty of entertaining and not practical solutions (Score:3)
I really did get a kick out of some of these responses. I sell data protection products for a living and 20TB is what I would consider an average small/medium customer. Every business these days has tens of terabytes of data. Of course they all need to backup their data, so there is nothing novel here. We have plenty of customers backing up hundreds of petabytes of data. Every dataset just needs a plan for backup, pretty simple.
The way I see it, this guy has a few options. One option is to just get more disk and make redundant a redundant copy. This would have have saved him in this case of the mistakenly erased raid, depending on how smart his sync script is. But a redundant copy is not a valid genuine backup plan. So many types of failures will show the holes of the dumb redundant copy.
The other option for a home user who's not looking to spend a bunch of money, is LTO6. They hold a sufficiently large amount of data, so only a handful of tapes will be needed. LTO6 drives are cheap enough, they won't break the bank. Since the data is on tape, you can shuttle the tapes to an off site location. Seems pretty simple.
FlexRaid or UnRaid (Score:3)
http://www.flexraid.com/ [flexraid.com]
http://lime-technology.com/ [lime-technology.com] (UnRaid)
Best solution for big media collections.
All data is stored seperatly on each drive, and 1 separate parity drive can protect up to 21 drives (as long as its as big or bigger than any 1 of those 21 drives).
Re: Don't bother. (Score:2, Insightful)
Exactly, do you really need to hoard all that content?
Re: (Score:3)
Re:Another RAID? (Score:4, Funny)
Re:Glacier at $20/mn expensive? (Score:4, Informative)
Glacier at $20 per month for 20TB is rediculously cheap by today's standards. And at those sizes, you'd want to ship those drives to Amazon instead of uploading. We do this all the time and it's not that hard.
The price of TBs of storage of course will come down without question. But by today's standards $20/month for a medium that won't "bit rot" on you is an amazing deal.
You missed a 0, he has 20,000GB and the cost for glacier is $.01/gb/mo (not including upload charges). So, Glacier would cost him $200 a month or $2400 a year. Not hugely expensive but if you are OK with a quasi-local copy (offline and stored in a fire safe, perhaps) you could do it cheaper for less, after you hit the 1 year mark.
Re: (Score:3)
Even with bandwidth, there are caps and fees here in the US. Try moving 1TB of data via LTE, and the telco will likely hand the person a five digit bill next month. Do it on some cable company plans, and you will be greeted by a $300 bill. So, large data via the Internet isn't going to happen.
There are a number of solutions for this problem:
1: One of the better ones is a server with decent backup software and a LTO tape drive. Then eight tapes will save the 20 TB. Expensive, but the job is done right.
Re: (Score:3)
The benefits of tape are:
Data will probably last 30 years (I have read 30 YO tapes myself) HD interfaces go out of fashion every few years.
You can have a pool of tapes, and recycle them when you no longer need the data.
Tapes will survive serious abuse. (A lot more than HDs anyway) definitely included the back of a station wagon (except in tropical climates).
You can use Am
Re:Go on the internet and find a DLT drive (Score:4, Informative)
You're dating yourself. LTO-5 is 1.5TB native, 3TB compressed at $25 per tape. LTO-6 is 2.5TB native and 6.25TB compressed. Both of those compressed numbers are using the built-in compression in the drive.
A 10-pack of LTO-5 tapes is about $250.
You can easily encrypt the tapes and tape them offsite. You can keep a copy onsite and offsite. You're simply not doing that with disk.
Your speed is also off - an LTO-5 can write at 280MB/sec. The limiting factor is not the write time on the media but the read time from disk.
Restore times are typically limited by the write rate on the destination raidset, not the read rate from tape.
Re:Go on the internet and find a DLT drive (Score:4, Informative)
LTO-6 can hold 2.5TB per tape, a tape cost ~$70, the drives cost $2000. That's still more expensive then just more HDDs for 20TB, but at >50TB it might be worth it.