Ask Slashdot: How Do You Manage Your Personal Data? 414
New submitter multimediavt writes "Ok, here's my problem. I have a lot of personal data! (And, no, it's not pr0n, warez, or anything the MPAA or RIAA would be concerned about.) I am realizing that I need to keep at least one spare drive the same size as my largest drive around in case of failure, or the need to reformat a drive due to corrupt file system issues. In my particular case I have a few external drives ranging in size from 200 GB to 2 TB (none with any more than 15 available), and the 2 TB drive is giving me fits at the moment so I need to move the data off and reformat the drive to see if it's just a file system issue or a component issue. I don't have 1.6 TB of free space anywhere and came to the above realization that an empty spare drive the size of my largest drive was needed. If I had a RAID I would have the same needs should a drive fail for some reason and the file system needed rebuilding. I am hitting a wall, and I am guessing that I am not the only one reaching this conclusion. This is my personal data and it is starting to become unbelievably unruly to deal with as far as data integrity and security are concerned. This problem is only going to get worse, and I'm sorry 'The Cloud' is not an acceptable nor practical solution. Tape for an individual as a backup mechanism is economically not feasible. Blu-ray Disc only holds 50 GB at best case and takes forever to backup any large amount of data, along with a great deal of human intervention in the process. So, as an individual with a large data collection and not a large budget, what do you see as options for now (other than keeping a spare blank drive around), and what do you see down the road that might help us deal with issues like this?"
Keep a spare blank drive around (Score:5, Insightful)
Re:Keep a spare blank drive around (Score:5, Informative)
Agreed. I've been gradually rotating larger backup drives in and smaller backup drives out over the last 10 years or so. Right now I have about 2 TB's of unique data in my archive which is kept on the host machine if it is regularly accessed or duplicated on another external hard drive. Everything (I care about) has two copies at all times. As my archive grows, I'm going to have to upgrade my archive device's capacity, but that's a given, no matter what you do, if you want it stored locally, you'll have to add capacity somewhere obviously. DVD-R's and BluRay discs aren't a viable option in my opinion, because I've got a ton of old self-burned discs that I recently had to toss because they were rendered useless from laser rot, even though they were in sealed containers in a cool, dry place.
The cloud is, to me, not a backup solution. I see it as a way to globally access my data and I use it as such. No sensitive data of mine will go to the cloud because the likelihood of needing access to it without warning is completely nil, so in my case, it's limited to media that I want constant access to. Now, the cloud definitely has the potential to serve as a backup solution, don't get me wrong, but there's just too much uncertainty involved in the cloud these days, especially as concerns the government nuking sites from orbit without warning, whether justified or not [slashdot.org].
However, I agree with some others that are telling you to do some house-cleaning. I recently went through my backups and found 300 GB's worth of crap that I hadn't accessed or used dating back to the early 2000's that I was saving for some stupid reason. Disc Images for ancient games that don't even run well on modern systems (or require a lot of fucking hassle to get running well), music that I haven't listened to in half a decade, old-ass videos that I'd downloaded from the internet back before there was such a thing as youtube, etc. Not to say that everyone's data is as silly as mine was, but it just added up over the years...
Use a NAS with backup (Score:5, Interesting)
We're up to 2 NAS units now, with 7TB[*] of disk space between them, all backed up on schedule. The USB backup drives are rotated every few weeks with another set kept in a secure place in the garage.
[*] One NAS unit doubles up as media server, so it's got a load of movies & music in addition to user files in its 6TB. The other one is our web server and email server with only 1TB of disk space.
Re: (Score:2)
http://www.newegg.com/Product/Product.aspx?Item=N82E16817707228 [newegg.com]
HDDs are kinda cheap. Keep one drive running and from time to time, run off a copy of the storage drive and slap a label with a date on a backup target drive. Do a daily task that checks the health of the HDD... automate it and send anything other than "green/healthy/pass" to yourself in email so you know when it's time to duplicate your main drive to retirement.
I love those HDD duplicators. They don't care about your OS and make perfect copi
Re: (Score:2)
Re:Keep a spare blank drive around (Score:5, Informative)
Right. Other than buying new disks, there is no good solution.
The asker seems to be looking for some kind of "join all my small disks together" solution. And yes, he can do this. RAID-0 or LVM. But... don't do it! If even one of those disks fails all the data is effectively gone. The solution is cheap to implement but totally worthless. Sorry, your 250Gb SATA disk now belongs in a museum.
RAID-5/6 is, IMO, also a bad idea; there are too many instances where the controller has failed or multiple disks have failed.
The asker explicitly excludes cloud solutions. It's depressing that people have recommended various cloud solutions nonetheless. Apart from not being answers to the question, these solutions are totally awful for large quantities of data. Amazon S3 may be nearly free if you want to store a few gigabytes, but if you want to store a few terabytes you are going to pay through the nose, and all the other service providers are the same. 2Tb would cost $234 per month just for storage, transfer cost not included. For the price of two weeks of S3 storage you can buy a 2Tb external disk. For the price of upload, download and a month's storage, you can buy four or five such disks and have as much redundancy as any normal person could ever need.
Re:Keep a spare blank drive around (Score:4, Insightful)
FYI: storing hard disks in a fire-proof safe is not a good idea. Fire-proof safes are generally rated for their ability to protect paper documents from burning up - but paper is very robust, and stable to very high temperatures provided it isn't actually exposed to a source of ignition.
This isn't really true of a hard disk - you can heat paper to 150 degrees C no problems, but as far as I know most hard disks when in storage may not actually survive prolonged exposure to those sorts of temperatures.
Re:Keep a spare blank drive around (Score:5, Insightful)
2) Each buy enough HDD's to duplicate your stuff
3) On a regular basis trade drives, update backups, trade back
4) If you are worried about security (either from them or someone breaking in), encrypt the drive(s) and keep one copy of the key with yourself and another in a safety deposit box (or another friend, etc).
Re:Keep a spare blank drive around (Score:4, Informative)
FWIW, my strategy is to keep truly important stuff on a raid enclosure (and backup to other disks periodically), and to just live with the fact that there's really nothing irreplaceable about the rest.
Re: (Score:3)
Another option is a couple of external drives that you keep at the office or a friends house, or the bank.
Re:Keep a spare blank drive around (Score:5, Informative)
Re:Keep a spare blank drive around (Score:5, Informative)
Link contains a referral ID, so Shikaku is earning from this, but not willing to say so.
Eventually, it ends up at http://www.backblaze.com/ [backblaze.com]
Solution.. buy hard drives! (Score:3, Informative)
1. Buy hard drive from brand A
2. Buy hard drive from brand B
3. put in seperate esata enclosures
4. backup to both drives.
Re: (Score:2)
Exactly. If you can't afford two new drives from different vendors large enough to hold your data, then you cannot afford to keep your data safe.
Don't bother fiddling with RAIDs unless you have many terabytes of data. Single drives are a lot faster to get and use.
Re: (Score:2)
Re:Solution.. buy hard drives! (Score:5, Informative)
Re: (Score:3)
Agree with this, mod up.
Particularly for home users (which TFA seems to indicate is the case here) a simple mirrored RAID array will do the trick. I recommend the following setup:
Buy 4 2TB drives.
Put 2 drives in a Mirrored array using motherboard-based RAID.
Put 1 drive in a USB 2.0, 3.0 or eSATA drive enclosure and back up RAID array to this drive.
Keep 4th drive as a spare.
Replace all 4 drives with larger drives as needed and available.
Done. You will almost never lose data using this method. If you REAL
Re: (Score:3)
Re:Solution.. buy hard drives! (Score:5, Interesting)
Honestly, it's my Western Digital drives that have lasted the longest. My dad is still rocking several single digit GB capacity WD drives actively in his legacy tower, and I've yet to have one die on me. Not to say I haven't replaced them as their capacity becomes outdated, but I've had much better luck with them than Maxtor (the worst brand I've ever used), which is now a part of Seagate, which I've also had a couple fail on me (but nowhere near as bad as Maxtor).
I've never used Hitachi or Samsung or any other brand that I know of, so I can't speak as to their quality, but I'm sticking with Western Digital.
Re: (Score:3)
For my own part, Maxtor, Quantum, and WD drives are the only consumer drives I've had fail on me. Have a dead 1TB caviar green drive on my desk right now. Then again, I haven't had to use any of the Seagate 7200.11s (or was it the 12s?) that had so many problems.
At least from the perspective of SMART output, the most reliable drives per hour put on them that I've run are Samsung drives, but those have also been a minority of drives I've used.
That said, I try not to put drives from one single manufacturing b
Re: (Score:2)
Re: (Score:2)
http://www.jwz.org/blog/2007/09/psa-backups/ [jwz.org]
Re: (Score:2)
Re: (Score:2)
RAID array on a spare box (Score:2)
Re:RAID array on a spare box (Score:5, Interesting)
I also use Dropbox for some stuff.
Enjoy your delusion (Score:4, Informative)
Re:Enjoy your delusion (Score:5, Insightful)
It's great that you know how fast his connection is and exactly what data restrictions his ISP imposes. I'm actually rather impressed you can be 100% sure his computer is connected to the internet at all. All I know is that if I had that much data, the time it would take to upload would probably be longer than the time it takes for the HDD to wear down and implode.
Re:Enjoy your delusion (Score:5, Funny)
I'm actually rather impressed you can be 100% sure his computer is connected to the internet at all.
Well he did post his question to an internet forum...
Re:Enjoy your delusion (Score:4, Funny)
Re: (Score:3)
http://www.wired.com/gadgetlab/2010/03/32gb-microsd-card/ [wired.com]
weight of a micro sd card :~ .5 grams
capacity of a micro sd card:~ 64 gb
http://www.amazon.com/SanDisk-Mobile-MicroSDXC-Memory-Adapter/dp/B005V7WIA2/ref=sr_1_1?ie=UTF8&qid=1332714352&sr=8-1 [amazon.com]
MATH:~ 1000/64 = 15.6 or 16 cards
https://www.google.com/search?q=1000%2F64&btnG=Search [google.com]
16 cards weighs 8 grams grams
Capacity of a pigeon:~ 38 grams
http://interbug.com/pigeon/technology/homing_pigeon_with_gps.pdf [interbug.com]
"Thirty-eight grams total is still a lot fo
Re: (Score:3)
Put them all in a coconut and you could transport them by swallow.
Re:Enjoy your delusion (Score:4, Insightful)
In typical "I need IT advice, but I have preconceived notions about how things should work and am not willing to budge on that" fashion, the asker has discounted some reasonable options without specifying the reasons that won't work for him, and failed to provide some super useful info like how large his data actually is, how often it changes, how much existing data changes, how much new data there is, and how quickly it grows.
So it could be that the reasons for his concern are unmerited, and GP merely points out that if his concern is privacy, there's ways to use the cloud safely. In typical Slashdot fashion, you rebuke the potential shortcomings of the advice without knowing whether those shortcomings actually apply to the asker.
Backup should be provided in depth, several prongs provides the best redundancy and the least single points of failure. Cloud storage is an excellent option for one of the prongs given certain factors. If most of the data rarely changes (pretty typical for very large data sets), incremental bandwidth usage past the initial storage is usually not much more than the data growth rate. As observed, it can be done in a way that respects privacy and safety.
Cloud storage has two main advantages over local backup solutions. You won't run out of disk space, and it's off-site (so a house fire won't take out your data set). Any on-site solution automatically fails that level of redundancy. Storage on S3 is ridiculously inexpensive any more.
I have about 6 TB of data that I need to keep backed up. I have about 12 years of digital photography and video originals, including stuff like wedding and honeymoon photos, as well as the birth and first years of my children's lives. When people suffer house fires, one of the most common and greatest laments are the things that can't be replaced - usually photographs.
My solution is four tiers. I have a local RAID0 in my Mac Pro. I have Time Machine backups of that (this is hands-down the best consumer on-site backup solution on the market). I rsync those files to a local RAID10 NAS device (Synology are a bit pricy, but they are completely worth it, really excellent built-in software with a lot of features you might find surprisingly useful, and you can purchase expansion bays to extend capacity as you're running low). Then finally I back up to Amazon S3 in encrypted form with JungleDisk (I no longer recommend this software, I own a copy of it from before it was bought by RackSpace, the quality has gone down since RackSpace bought it and "improved" it, plus I gather you now have to pay a monthly subscription, AND pay for your own storage - crap).
The only way my data is in jeopardy is if my house burns down (takes out 3 local redundancy & backup solutions) on the same day that Amazon has critical failure. And it's all 100% automated, Raid0 happens at time of write, TimeMachine alerts me if there's problems creating a backup and gives me local history, my NAS warns me by email & SMS if it so much as writes too slowly (my rsync cron script emails me if it can't reach the NAS for some reason), and JungleDisk does a nightly sync with S3, and sends me weekly reports so I can be sure that it's doing its job. I have quick local access, and slow offsite access if everything else fails (I'd probably go bum my work's huge pipe to do the initial restore if I had to rely on that).
Re:Enjoy your pricy delusion (Score:4, Insightful)
"Storage on S3 is ridiculously inexpensive any more.
I have about 6 TB of data that I need to keep backed up."
So you mean that 6000/month*0.125$=750$/month is cheap?
Or did I miss something?
Re: (Score:3, Insightful)
My solution is four tiers. I have a local RAID0 in my Mac Pro. ...
You do understand what RAID0 is, right? RAID0 is strictly for performance and offers zero data redundancy or failure protection. In fact, since you need both disks to function to read your data - you're essentially halving the MTBF of using one disk. Perhaps you meant RAID1? (a mirrored set)
Re:Enjoy your delusion (Score:5, Insightful)
Did you actually use it for recovery?
Both my rotated TimeMachines were corrupt. They never complained during backups, but failed miserably while trying to recover my Pictures HDD.
Only some of the backup files were corrupt, but when you try to recover a complete disk with TM, it's all or nothing, and the process stops after the first error, leaving you in the dust.
I had to write a parsing script with ruby, "cp -avX", ditto and chmod in order to get my system back.
It wasn't so hard, but it sure was stressful with one disk down, two corrupt disks and no other backup to get my pictures back.
BTW, TimeMachine doesn't backup every file in your system, and is too stupid to realize that it should not begin from scratch after recovery : it needs twice the storage after that, because it thinks every file is new.
My drives weren't big enough, so I had to wipe the backups and lose the local history.
Fuck it. I began using Carbon Copy Cloner since then, and never looked back.
It's free as in donationware, it works, it gives you a bootable backup that you can actually test and rotate properly, it can easily be automated, it archives the files that you've deleted between backups, and uses much less space than TimeMachine.
I hear SuperDuper is just as good.
TimeMachine is some crappy software with nice looking interface that gives you a false sense of security.
Re:Enjoy your delusion (Score:4, Informative)
S3 storage for 5TB isn't what I would call cheap. We are talking about $580/Month(or almost $7k per year). For that amount of money, you could buy a new set of 5TB worth of hard drives each month and then ship them to a remote location and pocket about $200 a month in savings.
Not a perfect solution(no online access) but I think it underscores just how costly S3 still is for large amounts of data. If you are talking about a few hundred GB of data, S3 storage is cheaper(and better) than anything you could reasonably do yourself. But once you scale up the usage... Heck, you could buy and colo a remote server and ship drives back and forth for less than what S3 would cost...
Re:Enjoy your delusion (Score:5, Informative)
You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.
Cloud is very trendy right now, but that doesn't mean it's a one-size-fits-all.
Re: (Score:2)
You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.
This doesn't prevent the OP from using local backup in the meantime. I backup to local storage as well as cloud. The local backups complete quickly in case I need to retrieve a file, and the cloud is there for if my house burns down.
The OP stated in his question that he has a lot of data but no money to buy redundant storage -- well, that's his real problem. If you have 3T of data, buy 3T of backup. I don't know what the OP is looking for other than a magic compression answer or something.
Re: (Score:2, Informative)
You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.
Cloud is very trendy right now, but that doesn't mean it's a one-size-fits-all.
Crashplan has an option where they will send you a hard drive to seed your backup locally and mail it back. That way you only have to do incremental backups once you do the initial seed.
If there's no offsite backup, the whole scheme is worthless. What happens if there is a fire?
Re:Enjoy your delusion (Score:5, Informative)
You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.
CrashPlan will let you Fedex them a hard drive to get the backup started. From then on, you only need to send deltas.
Try free, open-source SparkleShare (Score:2)
SparkleShare looks and works like Dropbox, but is actually just a fancy automated self-hosted GIT repo, (which you can interact with using GIT commands on a remote repo if that is what you want to do).
The wiki explains how to encrypt things (and the encfs recipe doc'd on the wiki also works with Dropbox, etc.)
I think the project has matured really well, but still isn't really well-known, and doesn't even get mentioned much on the slashdots, although that's where I heard about it.
www.sparkleshare.org
https:// [github.com]
Re: (Score:2)
"... there are cloud backup solutions that encrypt your data *before* it leaves the machine. ..."
That's nice, but there are some other issues. First, a major one: he has too much data for that -- it seems to be measured in terabytes. Since he's complaining that blue-ray is too slow, his upload speed will likely make backups up to the cloud an impractical solution also.
Second, there are some minor issues: cost and reliability. Even if he has plenty of upload bandwidth, having some cloud service hold on to that much data for you is not going to be cheaper than buying the extra hard drives yourself. Af
Re: (Score:2)
How do you know they don't have the decryption keys at the destination?
Because I'm using my own private key to encrypt the data. Research is your friend: http://support.crashplan.com/doku.php/articles/encryption_key?s%5B%5D=encryption [crashplan.com]
Re: (Score:3)
"Until that company goes out of business"
So what? It is not live data but one for backup purposes. If the company goes out of business you look for another one, no data is lost in the meantime.
Bare Drives and a USB Drive Dock? (Score:5, Informative)
One way to save a bit of cash is to buy a USB eSATA drive dock (single or double) with some bare eSATA drives. This cuts the enclosure out, and allows you to buy bare drives, which are often cheaper than enclosed drives.
You could also consider Drobo or one of the Wiebetech multi-drive RAID containers. But encryption + cloud isn't all bad.
Budget (Score:5, Informative)
"large data collection and not a large budget"
This is your problem right there. You can't enter into a a situation like this without planning a budget for the inevitable failures. I suggest purchasing a new larger drive (3TB are common now) and migrating the data from the problematic drive. Then migrate the data from several older smaller drives. This will reduce the component count (points of failure), save you power (cost in the long run) and keep you ahead of failures. You should plan on doing this periodically to maintain the integrity of the data.
Re: (Score:2)
Re: (Score:2)
The old smaller drives could go in a dedicated NAS box using software RAID. Doesn't have to be on all the time, just for periodic backups.
Look into MultiPAR for redundancy for your most important stuff too.
Re: (Score:3)
Yeah, in my view, there's a simple time-tested method for managing your data: Get a centralized storage medium big enough to hold it all, move everything to it, and then back that up. Your backup medium should be at least twice as big as your storage medium-- for example, if you have a 2TB drive that you're storing everything on, you should have a backup device that can hold at least 4TB.
But it seems like he's saying, "I want to store everything and back it up, but I'm unwilling to pay for the storage med
Buddy NAS (Score:5, Interesting)
I have a solution I call the "Buddy NAS". Go out and get two cheap computers. It could be a PC or a mini-NAS or a low-end server. Anything that will hold multiple hard drives. You jam both full of hard disks and use them as a backup/NAS server. One PC is kept at your place, the other at your friend's house.
Both computers have an account for you and an account for your friend (it helps if your friend is nerdy and "gets" backup solutions). Both of you now have a backup solution in your own home and a remote backup server at a friend's place. Two copies of your data, one remote. Basically it's like having local and cloud storage for you and your friend and it'll cost less than a grand if you shop around. If neither of you have static IPs you can use dyndns.org to connect to the remote boxes. Bandwidth shouldn't be an issue if you use rsync to backup changed files nightly.
Re: (Score:2)
I wish I could mod that up, its actually a pretty decent solution with lots of potential for win.
Re: (Score:2)
I see one little problem, these computers are Always On' and will fail sooner than a disk that's connected once a day or once a week.
That's why I have two large HD's, one at home and one at the GF's place, we rotated them about once a moth.
One spare isn't enough (Score:2)
I have invested in USB backup drives of about the same total capacity as my primary storage drives. Yes, that's a lot of hard drive space for backups, but it's really the only practical solution that I have found. Just think of it as the cost of not losing all that data to the inevitable drive failure.
An external eSATA drive dock and a stack of 2TB drives might be a somewhat better way to go about it, at least backups and restores would be faster than the USB drives.
Here's a couple of solutions (Score:3)
I can offer a couple of suggestions... What I did was buy a used Dell Poweredge 2950 on eBay for about $500 bucks shipped and I added 4 x 1tb SATA drives to it and I run a raid 5 setup with 3tb of usable space across the four 1tb drives. This solution cost me less then $1000 and I have a nice playground to experiment with VMWare ESXi.
I know that's not exactly budget conscience but it works great for me.
If I were on a tight budget I would just buy a 2tb USB drive from Newegg or somewhere similar. It looks like you can buy a name brand for about $130 bucks.
If you have a little bit more money to spend you could always buy a couple of 2tb internal SATA drives and run RAID-1 mirroring on them. You could put these into an old computer and make a little NAS linux server...
If you're saying you have no money to spend then maybe you need to consider cleaning up your data. Often times all those "personal files" that you think you need to keep... Really aren't required. Just my 2 cents but this problem is very solvable.
I delete stuff (Score:5, Insightful)
Is your time more valuable than a new disc? (Score:4, Insightful)
Obviously you still have an issue of tracking things down on the rare occasions when you actually need some of your family photos. But you can rest assured that they're in there somewhere and weren't purged last time you needed a few GB for more webserver logs.
Maybe the first step is to de-dup the existing data. You'll still have some manual intervention to check possible duplicates, but it's a first step towards tackling the bigger problem.
Re: (Score:2)
Re: (Score:3)
Obligatory FreeNAS comment... (Score:2)
Cheapo used market PC, invest in some large drives and a couple of drive docks, install FreeNAS.
Take a weekend to organize your data however it makes sense (by year, subject, file type, whatever), and store it on a particular drive. Rinse wash and repeat. Depending on how important the data is, store in a fireproof safe onsite, or offsite. When (read: if) you need the data again, dock the drive and retrieve.
Personally, I'm about to liberate myself from years of data. I'm tired of all these bloody drives
two maxims (Score:2)
One, you can't have enough backup images as something always seems to go wrong. Should include at least one offline unplugged "safe" unit. I know, it's a hassle to keep them up to date.
Two, the longer you wait the less all this backup space costs, so don't buy too much too soon.
Magic (Score:5, Insightful)
So your disks are full and possibly broken. You don't want to have more disks, you don't want tape or optical medias, and a storage provider (aka The Cloud) is not an option... Then you have three solutions "down the road":
1) Delete stuff
2) Invent a new compression algorithm that will allow you to reuse the same disks forever without losing data
3) Rely on magic*
*might overlap with solution #2
Re: (Score:2)
1. data deduplication. Most people have a lot of redundancy ever since hard drives started growing past a few gigabytes.
2. for photos - people might have 20, 30 shots, of which only 2 or 3 are really "keepers." Get rid of the crapola. Same goes for videos that are "better off forgotten" or that, while still viewable, wuld be a pain to watch because we no longer can stand looking at 320x200.
3. tar lots of little files into one big file, so as to not lose disk space to 4k sectors holding a 3
Data compression... (Score:2)
It depends on the data, but many formats compress really well when using WinRAR. Many of my files, for example, that reach nearly 10:1 compression. Unless we are in the same profession, I wouldn't set your expectations that high, but I imagine on average you could get your data usage down to 40%. If I'm right, maybe you could winRAR several folders from the failing drive to the smaller drives, and not necessarily need to get more space available.
That said, I really do think the suggestion of buying an
Re: (Score:2)
Now I spend many an evening studying and devising new means of compression by which I may squeeze a tiny bit more useful information into those bits. I'm good at it, too.
Also... in my very extensive testing experience, though winrar beats zip with ease, the contest is close between rar and 7z on small files - and once the files get big, the advantage of 7zs -md=256m dictionary size gives it a clear edg
Redundancy and Archiving. (Score:2)
As several people have said you already answered the question yourself. Spare HDD + Blueray.
You can achieve what you want by also changing the way you think about your data.
How much of your personal data is live? As in, how much of it do you access constantly, and need immediate access to?
Here's what I do, I have discrete HDD set up for each data type (not needed...but I had spare ~500gb drives so it's how I did it) There are broken down to Music, Projects, Video, and Photos. Each of them is synced mon
What kind of data? (Score:2)
You must give up some of it, or transfer it to some other, long-lived medium.
Otherwhise, I suggest you face reality and invest accordingly [youtube.com]
Drobo + BackBlaze = Win! (Score:4)
Drobo -> mostly reliable local backup
BackBlaze -> mostly reliable offsite backup
You might want to substitute a ZFS-based FreeNAS for the Drobo, if you're so inclined. It's less automatic, but seems just as reliable.
Depends what you mean by "personal" data (Score:2)
All of my personal data is in my home directory and easily backed up to non-volatile media (which I do a few times a year, but not as often as I should.)
All of the project data is on SourceForge or company project servers, so there are duplicate copies of that.
I hardly think of my music or movies as "personal" data nor as irreplaceable. Were I still playing video games, I don't think I'd bother backing up game data, either.
When people talk about needing entire drives for their personal data backups,
Just delete it! (Score:2, Insightful)
Fire safe + USB drive (Score:2)
My fileserver uses RAID and makes a separate (encrypted) backup to an external USB hard drive (fortunately, my data hasn't grown faster than hard drive sizes so I can fit it all on a single 2TB drive, to ensure file integrity, periodically I have rsync verify file checksums,)
As a secondary backup, I use a 1TB notebook drive locked in a USB enabled fire safe:
http://www.amazon.com/SentrySafe-QA0121-Fire-Safe-Waterproof-Storage/dp/B00166187Q/ref=cm_cr_pr_product_top [amazon.com]
I used metal straps to tie it down and lock
Have Less Data or Build a Server (Score:3, Insightful)
I think it's time to admit that you're a hoarder. What exactly -is- your personal data that's so precious? I run a server just to keep my skill set up and run my side business, but I've only managed to accumulate around 600GB of data, only about 35GB of it is 'mine', the rest is client backups.
So first admit that you're a hoarder, then decide if you wan to address that issue or indulge it. If you choose to indulge it, you're going to want to build a small home server. Something with a low-end 64-bit CPU (i3?), a gigabit LAN port, and lots SATA ports and 3.5" drive bays. Buy a bunch of high-quality (WD RE4?) matching drives that fit your data needs times two (you're RAIDing space away). Once you have that, install Linux on it, build a software RAID-1 or 0+1 array (don't do RAID-5 unless you can handle days of rebuild time), and format it with something accessible (read: in the kernel, like EXT4). Create a share on the array with Samba and happily access it from all your machines (don't bother with Netatalk or NFS; CIFS is great on all platforms). As your data needs grow, you can add drives in pairs or replace drives with larger ones and grow the volume. If you need backup, you'll want another array, preferably on another low-end box (an enclosure on your desktop?) but it can be built on a RAID-0 or JBOD to save money.
It's that time again, is it? (Score:5, Insightful)
It's that time again, is it?
http://ask.slashdot.org/comments.pl?sid=2452630&cid=37557630 [slashdot.org]
Either.. ..or..
A: Buy that HDD. Yes, they're a bit more expensive right now
B: Wait a few months, prices will come down again, buy that HDD then. Yes, you may lose your data in the mean time.
Now stop asking or I'm going to pull over.
Re: (Score:3)
Don't worry - happens to the best of us; I just think that /. should do better in these stories in terms of filtering them (off the front page or entirely). But that's a different discussion :)
Your question seemed to set limits which are unrealistic, that's why the conclusion is really 'HDD' even though you specifically set that as something that wouldn't be an option.
I can tell you how I do it, it's just not cheap. Also, it's not really 'managing' my data.. it's a storage/backup solution. The difference
So you do not have backup? (Score:2)
Drives die, sometimes without warning (and old statistic by IBM says 50% of the time there is no warning). You could just throw everything away, as you are going to lose it anyways, sooner or later. Or you could find the resources needed to make sure you have everything on at the very least two drives (one of which should not be connected or running). There is nothing that can replace reasonable backup.
As a side note: Common sysadmin wisdom is to have 3 independent backups in addition to the original.
Hoarders: Digital Edition (Score:5, Funny)
You might be on the next spinoff of Hoarders programs, a digital hoarders show.
In this show, redundancy, old versions, and files that haven't been opened in 5+ years are brought into question, for which you will be embarrased to defend... You will attempt to justify why you still have linksys drivers for a wrt54g you don't even have anymore. And no, the DVD ISO of the Alvin and the Chipmunks movie, that you never burned or watched, is not worth saving.... Neither are about 85% of the digital pictures you took (you know, the ones that were the 'bad shot' that you took before finally getting the good one).
Take a day or two, go through it chunk by chunk, and purge! PURGE!
Re:Hoarders: Digital Edition (Score:4, Informative)
You nearly said it, but I'll just come out and ask: what is this vast quantity of personal data? If it isn't downloaded movies or rips of DVDs the OP owns... Maybe he's a film maker or prolific but unpublished musician.
We need more info because the only option, other than deleting stuff and throwing money at the problem, is some clever solution based on the specific data in question.
Raid + mirror with a friend (Score:2)
A friend and I each have FreeNas servers with multi TB raid-z.
Some of our data we keep mirrored between them.
The servers are physically brought together occasionally for a full sync, but most of the time rsync -n is done over the internet to see what needs to be updated and the data transferred on a removable hard drive.
That's an easy one. (Score:2)
Storage? (Score:2)
Floppies
More floppies
More floppies
(I can sell you some cheap!)
Budget? (Score:2)
As others have noted, if you can't afford AT LEAST another drive, serious problem right off the bat. One wonders what the data is worth given this.
I'll move on, assume the data is worth AT LEAST another drive or two (we're talking a couple hundred bucks at most, come on):
1. drbd: raid to a low cost, remote machine with similar sized drive. Dead drive is now recoverable.
2. amanda or similar backup to drive on remote machine. No, not tape, just virtual on disk. Now have a backup history as well in case one ne
Bite the bullet (Score:2)
Get older version LTO drive, get identical harddisks with spares and do full drive image backups with incremental backups in-between.
Mirroring disks (Raid 0+1) is good too.
All that other stuff - DVD/Blue Ray, cloud is chickenshit. Drives die or act up.
Run Linux on one drive and do image backups from there.
Zerofill the empty space,
dd if=/dev/zero | split --verbose -b 2000m -d - ZERO
compress the image:
dd if=$DEVICE | gzip -v | split --verbose -b 2000m -d - NAME
and write the chunks to tape.
Define what your Backup Must Do First & Why (Score:3)
People have different needs. Some needs are imposed by either employers or the wonderful US Govt. for mandatory data retention. Others are your life's design work that you want to retain until you die. Other data you want to pass to your kids. If you can't afford to lose it keep multiple backups on multiple media in multiple locations. Books & pamphlets have been written on this. Transfer the data to new media once a year or two or three & keep all working drives.
No single storage device local or remote is immune from disaster. The Alexandria Library succumbed and took with it countless early human treasures. Wars have done in archives all over the world. Lightning, outages and power surges can defeat the best protections even when electronic equipment is turned off, but still plugged in (laptops are better when left unplugged, which is actually a great asset).
Backup is one thing; recovery is another and it can be GUT WRENCHING. The recovery process needs as much thought as backup.
A Clue or Two: A business partner had his MBPro backed up to 2 external HDs. Not great, but OK. Said MBPro crashed on the Lion upgrade. No way to know whether it was hardware or software and the MBPro should have at that point been off limits for use until carefully checked out. He happens to live in an area subject to lightning and outages which can affect anyone (even with a UPS). However, he reinstalled the Snow Leopard and plugged the first BU HD in an attempt to reload the data; HD became corrupted. Should have stopped, but then the 2nd HD was corrupted. Moral of the story; Recover data from a backup to an external HD running on another computer than the one that got mucked up.
The cost of 3-4 external 2-3 Terabyte hard drives and a couple cases or RAID box is dirt cheap compared to the value of the hours you put in on your computer each year as are Blue Ray drives & disks.
Caution: Someone on this list mentioned putting drives and disks in a "fireproof safe" or "fireproof file cabinet"; wrong! The UL approved boxes are designed only to protect "paper" for a given amount of time in a typical fire by releasing steam (212 deg. F = goodbye DVD/BR disks). Once the fireproof agent uses up its water...Farenheight 456 takes care of all contents...permanently. This is why multiple locations are needed.
How important is uptime? (Score:2)
In that case, how important is uptime to you? Since this is personal data, I'm guessing that you could live without live access to it for a few days. And given that, I think your best bet is not to keep a spare just sitting around, but to only buy one when you need it. Hard disk prices keep going down, and the price for the sa
Quit screwing around. (Score:2)
SDLT tape drive and some tapes. If your "personal data" is not worth the $800.00 to buy a good used SDLT drive and a few tapes, then it's not worth backing up.
Just do not dink around with theoretical "backup solutions" that are not proven. and no, hard drives are not "proven" for reliable and long term backup. I have DLT tapes from 16 years ago that I know for a fact I can still read.
If your data is important, You dont screw around with consumer hard drives that are known to have a low MTBF.
Manual on External HDs and good organization (Score:3)
Good organization outweighs medium in my case. 2xExternal 2 TB HDs - primary and secondary...and then a third stored off site at my parents that I update about 3 times a year, so if the worst happens I'm 6 mons out of date, but its usually about 4. And thats if both my primary and secondary go down. Thats a cost of about $300 total and a little a bit of effort.
"A little bit of effort" is defined by how you organize. Backing up manually means I don't rely on software or a service, but it requires some forethought. For me I break it up by data type and usually year...sometimes I go one more by how that data was acquired (photos I add who took the picture). This is important because I put anything new into a diff folder so I know whats new and whats not. It took me a couple of years to get to the structure I have but I sometimes add small tweaks. The effort or time now is fairly miniscule.
What I'm trying to get at is this : if you're prepared to put a small amount of time in every now and again, with an initial overhead, you can do this very easily and cheaply.
Re: (Score:2)
Well let's see (Score:3)
Data Hoarding and my solution (Score:4, Interesting)
First, let's look at your problem: You are gathering too much data. Either the data is 100% needed and irreplaceable, or it isn't. If it isn't, your first step is to treat your data just like you would physical junk that accumulates in your house.
Create Three folders.
1. Critical Keep
2. Unsure
3. Toss
Go through your data and MOVE it to one of those three folders. If it isn't critically important data that you would be upset that you lost and can't be recreated (wedding videos, etc) It goes in the Critical Keep folder. If you aren't sure about it right now, but you can't declare it for folder 1, put it in 2. Anything else "old install files, backup data from a windows 98 machine, etc" That stuff can be deleted. Be harsh with yourself. Think of it like moving from house to house, if you haven't opened that box by your third move, just toss it in folder 3.
Repeat the process until you either have everything in your Critical Keep folder, or your delete folder.
Now, hopefully you have reduced the size of the data you are using to something marginally manageable. I'm a data hoarder, and I've managed to keep the rate of growth of my data to lag behind the general rate of growth of HDD capacity. Now for the fun stuff:
Two things you want to avoid.
1. Loss due to a dying disk
2. Loss due to a destroyed home (fire, theft, etc)
Here was my budget solution that resulted in a fire and forget backup system that is suitable for a home user and is about as minimal as you can get for cost.
3 Disk Drives.
A primary drive to run the operating system and hold installed programs and two LARGE data drives in a RAID1 configuration.
Static data files (Video, pictures, etc) get stored on the RAID1
A scheduled process (once per month for me) backs up the OS drive to a virtual HD file on the RAID1. The files on the RAID1 are then backed up to a cloud storage service (Carbonite in my case).
So, what is the result of this?
My operating environment is backed up monthly. The only thing I lose here is configuration changes or programs installed since the last backup (less than 30 days for me)
The RAID1 ensures that my personal/static data is protected from a single disk failure, and helps a bit with read performance for the static (and large) files.
Should a cataclysmic failure occur and my entire computer is lost to something like a fire, remember that I've been sending what is on the RAID0 out to the cloud (carbonite), so when I can rebuild a computer I can just download the (very large) offsite backup from the cloud to my new machine.
The downsides I have right now:
1. I maintain the windows backup as a VHD file because it allows me to ensure that the backup data is 'packaged'. I don't know the exact details about windows backup, but given that Carbonite sometimes excludes system files I didn't want to risk an important hidden/system file being missed in the backup. In addition I didn't like how it could only backup to the root folder of a drive. The downside is that the resulting 100GB file is a pain to backup, which is why I restrict the backup histerisis to 30days (previously I had it backup every 3 days) This keeps it from continually uploading the VHD file to carbonite.
2. The HDDs for the raid1 lose half their total capacity in that configuration. I used it because it let me only have to use 2 drives and the performance boost. If you can afford 3 drives, go for a RAID5.
3. Most Motherboards support RAID natively now. However, I understand that you can run into issues with hardware RAID if you have to switch to a different hardware solution. I haven't tested this, but it could potentially be an issue if you use a RAID5 from hardware and your motherboard fails and you can't replace it with an exact model. The good news here though, is if you have been backing up to the cloud, typically it's done on a per file basis, and thus you don't have to worry about this. Just download your stuff ba
My 2 TB errrr cents (Score:3)
Delete your porn
The rest of your personal data will fit on a floppy.
Fireproof is NOT Firesafe; Oxymoron (Score:3)
The words "Fireproof Safe" is by the definition of Underwriter Labs. It merely refers to being fire resistive for a given amount of hours in a typical fire "for paper".
Forget DVDs and CDs and any hard drives surviving a fire in one of these "fireproof" devices. They are designed to release steam to keep the temperature at 212 def F until the fireproofing material exhausts all its water at which time the temperature goes up and, well...you can imagine what.
Re:Fireproof is NOT Firesafe; Oxymoron (Score:4, Informative)
Unless, of course, you get the safe that's rated for computer media.
This is an area where you really need to RTFTS (tech specs) as it will tell you EXACTLY what kind of fire, temperature and duration it will protect a specific kind of media for.
rsnapshot + raid6 on server in basement (Score:4, Interesting)
My solution to this problem is painfully simple: about 5 years ago I bought 5 drives 500GB each. I have put a server (made from old parts, like pentium IV and so on) in the basement (where nobody hears it, and it can be as noisy as hell). I installed debian on it and configured cron to call rsnapshot three times per day for doing automatic backups of all PCs in my family. I never touched this machine since then.
With one exception: 3 years ago I started to run out of space, so I bought 2 HDDs 2 TB each, reconfigured raid6, which was extremely easy because for raid I am using mdadm, which supports such operations online. Also I had few more spare drives during the years, so I kept adding them to the array, and currently there are 9 HDDs in this PC. It is very noisy, but nobody cares about that.
It runs flawlessy, untouched for years, and nobody cares about it, except for when somebody in my family accidentally loses or deletes a file. Then suddenly backup comes very handy.
Rsnapshot is especially good, because it keeps hardlinked copies of data from last week, 2 weeks ago, last month, and much more, depending on how you configure /etc/rsnapshot.conf. Currently I have backups dating back about 2 years, with granularity of 1 month. And it only occupies the space on HDD to reflect the changes between data, thanks to hardlinks.
So my raid6 array has total size about 4TB and still 500GB free. And I feel this will last at least a year or two. In case of problems I can start deleting copies that are more than 1 year old. While most recent snapshot uses about 2 TB or such.
Rsnapshot also can backup windows machines, so you don't need to worry about compatibility. Though I don't have windows machines and I don't test that in practice ;)
So now EVERYTHING is powered? (Score:3)
And now there's 200 comments where the people are proud of their kilowatt server arrays which are powered 24 hours a day for their PHOTO ALBUMS? Are you people shitting me? I mean, you're putting me on, right? You don't really use up 10,000 kWH per year storing your family photos, do you?
Hey, I've just invented the electric elevator-button-pusher. I save a TON of finger wear and tear.
Sometimes, humanity makes me sad.
The delete button (Score:3)
Not trying to troll here. I'm serious. Consider that you might be simply storing too much stuff.
Ok, let me steer a little... (Score:3)
Data integrity (Score:4, Interesting)
Keep all your important files in a version control system. Personally, I use Perforce (it's free for 2 users or less). That gives you: multi-revision history and checkin comments, an easy way to pull a subset of files to any computer in your house, and peace of mind that you don't need to worry about kids deleting anything important as it's all stored on the server with history. Also easy to see what has changed on any computer and check those files in. And there's a big win for data integrity checks: Perforce stores the checksum of all files (and revisions) and can easily check that every file still matches the checksum in the central database. If you have any disk corruption, you'll know about it when you run 'p4 verify -q //...'. You can store files of several gigabytes each with no problem.
On top of this, I use rsync to copy the server data onto backup drives. I'm also looking at storing backups online, but haven't taken that step yet.
I've been using this system for years and I couldn't imagine being without it. It's so easy to find and retrieve exactly what I want - my resume 5 revisions ago, my tax return, photos from 2003. Even without that, the data integrity checks give a lot of peace of mind.
Re: (Score:2)
Except the latest batch of Synology machines (like my recent DS1512+) seem to have a hardware flaw. You reboot them and they die. And then just TRY getting Synology tech support over the weekend........NOT HAPPENING.
Raid and an NAS is a great thing. My Infrant (Netgear) NV has been solid for 5 some years now. Synology seems like a nice product, but if the thing is going to freeze up and die on a reboot and I can't get tech support for it over the weekend........fuck that.
Re: (Score:2)
- Rather than NFS, I use a tarpipe: Source box uses xinetd. It's a little faster, and uses less processor time at the backup box. Which is important, because...
- The backups box also compresses the data it gets (pigz or bpzip2, depending where the data is from) and encrypts it (gpg - have to use assymetric encryption so the backup box doesn't need to store the key) in case of theft. It takes rather a lot of processing power to handle this, but a Core 2 Quad does
Re: (Score:3)
I do consider tape the best answer to backups. No, it isn't flashy like cloud storage, or the latest Internet rendition of it (be it a glorified file share, rsync, etc.)
Modern tape drives are a lot more reliable than the old 8mms and QIC cartridges. I still have DLT tapes from '98 which are still readable.
It all depends on the size of data. For the amount the OP has, if there is money, I'd consider an external LTO tape drive. They are around $1400 on NewEgg for a LTO-3 external, plus one will need a cou