Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
Data Storage Hardware

Ask Slashdot: How Do You Manage Your Personal Data? 414

Posted by timothy
from the it's-2012-how-do-you-back-up? dept.
New submitter multimediavt writes "Ok, here's my problem. I have a lot of personal data! (And, no, it's not pr0n, warez, or anything the MPAA or RIAA would be concerned about.) I am realizing that I need to keep at least one spare drive the same size as my largest drive around in case of failure, or the need to reformat a drive due to corrupt file system issues. In my particular case I have a few external drives ranging in size from 200 GB to 2 TB (none with any more than 15 available), and the 2 TB drive is giving me fits at the moment so I need to move the data off and reformat the drive to see if it's just a file system issue or a component issue. I don't have 1.6 TB of free space anywhere and came to the above realization that an empty spare drive the size of my largest drive was needed. If I had a RAID I would have the same needs should a drive fail for some reason and the file system needed rebuilding. I am hitting a wall, and I am guessing that I am not the only one reaching this conclusion. This is my personal data and it is starting to become unbelievably unruly to deal with as far as data integrity and security are concerned. This problem is only going to get worse, and I'm sorry 'The Cloud' is not an acceptable nor practical solution. Tape for an individual as a backup mechanism is economically not feasible. Blu-ray Disc only holds 50 GB at best case and takes forever to backup any large amount of data, along with a great deal of human intervention in the process. So, as an individual with a large data collection and not a large budget, what do you see as options for now (other than keeping a spare blank drive around), and what do you see down the road that might help us deal with issues like this?"
This discussion has been archived. No new comments can be posted.

Ask Slashdot: How Do You Manage Your Personal Data?

Comments Filter:
  • by Anonymous Coward on Sunday March 25, 2012 @12:40PM (#39467313)
    I think you already have the answer
    • by AngryDeuce (2205124) on Sunday March 25, 2012 @01:04PM (#39467535)

      Agreed. I've been gradually rotating larger backup drives in and smaller backup drives out over the last 10 years or so. Right now I have about 2 TB's of unique data in my archive which is kept on the host machine if it is regularly accessed or duplicated on another external hard drive. Everything (I care about) has two copies at all times. As my archive grows, I'm going to have to upgrade my archive device's capacity, but that's a given, no matter what you do, if you want it stored locally, you'll have to add capacity somewhere obviously. DVD-R's and BluRay discs aren't a viable option in my opinion, because I've got a ton of old self-burned discs that I recently had to toss because they were rendered useless from laser rot, even though they were in sealed containers in a cool, dry place.

      The cloud is, to me, not a backup solution. I see it as a way to globally access my data and I use it as such. No sensitive data of mine will go to the cloud because the likelihood of needing access to it without warning is completely nil, so in my case, it's limited to media that I want constant access to. Now, the cloud definitely has the potential to serve as a backup solution, don't get me wrong, but there's just too much uncertainty involved in the cloud these days, especially as concerns the government nuking sites from orbit without warning, whether justified or not [slashdot.org].

      However, I agree with some others that are telling you to do some house-cleaning. I recently went through my backups and found 300 GB's worth of crap that I hadn't accessed or used dating back to the early 2000's that I was saving for some stupid reason. Disc Images for ancient games that don't even run well on modern systems (or require a lot of fucking hassle to get running well), music that I haven't listened to in half a decade, old-ass videos that I'd downloaded from the internet back before there was such a thing as youtube, etc. Not to say that everyone's data is as silly as mine was, but it just added up over the years...

      • by AliasMarlowe (1042386) on Sunday March 25, 2012 @02:41PM (#39468227) Journal
        What I did some years ago was recognize that "manual backups" were not done often enough, and important stuff was scattered around a few PCs. So I got a NAS [synology.com], stuck a pair of disks into it (RAID 0 for speed), and set up its automated incremental backup to run 3 times per week to an external USB drive. The PCs now mount the NAS at login, and that's where all data files are stored by default (even the kids use it).

        We're up to 2 NAS units now, with 7TB[*] of disk space between them, all backed up on schedule. The USB backup drives are rotated every few weeks with another set kept in a secure place in the garage.

        [*] One NAS unit doubles up as media server, so it's got a load of movies & music in addition to user files in its 6TB. The other one is our web server and email server with only 1TB of disk space.

    • by erroneus (253617)

      http://www.newegg.com/Product/Product.aspx?Item=N82E16817707228 [newegg.com]

      HDDs are kinda cheap. Keep one drive running and from time to time, run off a copy of the storage drive and slap a label with a date on a backup target drive. Do a daily task that checks the health of the HDD... automate it and send anything other than "green/healthy/pass" to yourself in email so you know when it's time to duplicate your main drive to retirement.

      I love those HDD duplicators. They don't care about your OS and make perfect copi

    • I have a Media PC on the TV in the living room, and a desktop, and a laptop. My e-mail is on all three. Most work type documents are on the desktop and the laptop. Bulk media is on the desktop and the Media PC. Total redundancy. (As long as I keep growing the drive space.)
    • by JackDW (904211) on Sunday March 25, 2012 @02:39PM (#39468205) Homepage

      Right. Other than buying new disks, there is no good solution.

      The asker seems to be looking for some kind of "join all my small disks together" solution. And yes, he can do this. RAID-0 or LVM. But... don't do it! If even one of those disks fails all the data is effectively gone. The solution is cheap to implement but totally worthless. Sorry, your 250Gb SATA disk now belongs in a museum.

      RAID-5/6 is, IMO, also a bad idea; there are too many instances where the controller has failed or multiple disks have failed.

      The asker explicitly excludes cloud solutions. It's depressing that people have recommended various cloud solutions nonetheless. Apart from not being answers to the question, these solutions are totally awful for large quantities of data. Amazon S3 may be nearly free if you want to store a few gigabytes, but if you want to store a few terabytes you are going to pay through the nose, and all the other service providers are the same. 2Tb would cost $234 per month just for storage, transfer cost not included. For the price of two weeks of S3 storage you can buy a 2Tb external disk. For the price of upload, download and a month's storage, you can buy four or five such disks and have as much redundancy as any normal person could ever need.

  • by FrozenFood (2515360) on Sunday March 25, 2012 @12:40PM (#39467317)

    1. Buy hard drive from brand A
    2. Buy hard drive from brand B
    3. put in seperate esata enclosures
    4. backup to both drives.

    • by Ichoran (106539)

      Exactly. If you can't afford two new drives from different vendors large enough to hold your data, then you cannot afford to keep your data safe.

      Don't bother fiddling with RAIDs unless you have many terabytes of data. Single drives are a lot faster to get and use.

      • by swalve (1980968)
        RAID is nice in that it helps with uptime. A drive failure doesn't mean you have to copy a bunch of stuff all over again. Just reboot, replace and let it resync on its own. It is also really nice to have all your storage in one giant blob of space, even if it isn't multiple TB (beyond the size of a single drive). On the other hand, I am comfortable with how RAID works and I find it easier to manage than a bunch of single drives and various copies. Not everyone might have the same comfort level.
        • by TheRaven64 (641858) on Sunday March 25, 2012 @02:00PM (#39467983) Journal
          If you're using ZFS, then the best solution is to use RAID-Z for online storage and then have two external disks which you use zfs send / zfs receive to update. This means that catastrophic failure (e.g. a power supply problem blowing all of the drives in the machine) will still leave you able to recover. Ideally, you should store one drive at home and one elsewhere, so that if someone steals your computer then they don't get the data.
        • by d3ac0n (715594)

          Agree with this, mod up.

          Particularly for home users (which TFA seems to indicate is the case here) a simple mirrored RAID array will do the trick. I recommend the following setup:

          Buy 4 2TB drives.

          Put 2 drives in a Mirrored array using motherboard-based RAID.

          Put 1 drive in a USB 2.0, 3.0 or eSATA drive enclosure and back up RAID array to this drive.

          Keep 4th drive as a spare.

          Replace all 4 drives with larger drives as needed and available.

          Done. You will almost never lose data using this method. If you REAL

    • My approach is to buy a few drives at a time of exactly the same model. I've had more electronics failures than mechanical failures - this way if a drive fails electronically, I can swap in the board from another drive to get the data off.
    • by zonky (1153039)
      Is this what counts as a backup? two drives in the same location?
  • I run a RAID5 array on a spare box for backups, totaling 8TB before file system and RAID takes out its chunk. It's only turned on during backups, and is a fairly cheap solution for lots of storage if you look for sales on drives.
    • by swalve (1980968) on Sunday March 25, 2012 @01:22PM (#39467701)
      That's not a bad idea. I started with the OP's problem, trying to keep data from multiple machines in sync and backed up and with enough room to spare. After having spent more weekends copying data back and forth to clear out a drive in order to replace it, I decided to go to the fileserver paradigm. I built a machine with three 40gb drives RAIDed together and made that the only place useful data would be stored. I've since expanded it up to 3tb in various increments, and it has worked well. It has saved tons of time and money by allowing my computers to use whatever cheap harddrive was available and just restore from backup when it went TU. But with the need for increased data availability outside my house (IE, making my notebook my main computer), I'm starting to reverse course and move to your idea. Using robocopy on the clients and shell scripts + hard links on the server, I've set up a workable versioning backup system that doesn't take up too much space.

      I also use Dropbox for some stuff.
  • Enjoy your delusion (Score:4, Informative)

    by Trixter (9555) on Sunday March 25, 2012 @12:42PM (#39467337) Homepage
    "I'm sorry 'The Cloud' is not an acceptable nor practical solution." Not sure what brand tin-foil hat you're wearing, but there are cloud backup solutions that encrypt your data *before* it leaves the machine. I use CrashPlan (I can't speak for others) and I've verified the encryption myself by capturing the traffic leaving my machine, even when CrashPlan was backing up to other machines on my own private network. Even the data it writes to locally-attached hard drives is encrypted. So there's at least one company who gets it right.
    • by Anonymous Coward on Sunday March 25, 2012 @12:46PM (#39467377)

      It's great that you know how fast his connection is and exactly what data restrictions his ISP imposes. I'm actually rather impressed you can be 100% sure his computer is connected to the internet at all. All I know is that if I had that much data, the time it would take to upload would probably be longer than the time it takes for the HDD to wear down and implode.

      • by AmiMoJo (196126) <[ten.3dlrow] [ta] [ojom]> on Sunday March 25, 2012 @01:38PM (#39467827) Homepage

        I'm actually rather impressed you can be 100% sure his computer is connected to the internet at all.

        Well he did post his question to an internet forum...

      • by Anonymous Coward on Sunday March 25, 2012 @01:43PM (#39467863)

        In typical "I need IT advice, but I have preconceived notions about how things should work and am not willing to budge on that" fashion, the asker has discounted some reasonable options without specifying the reasons that won't work for him, and failed to provide some super useful info like how large his data actually is, how often it changes, how much existing data changes, how much new data there is, and how quickly it grows.

        So it could be that the reasons for his concern are unmerited, and GP merely points out that if his concern is privacy, there's ways to use the cloud safely. In typical Slashdot fashion, you rebuke the potential shortcomings of the advice without knowing whether those shortcomings actually apply to the asker.

        Backup should be provided in depth, several prongs provides the best redundancy and the least single points of failure. Cloud storage is an excellent option for one of the prongs given certain factors. If most of the data rarely changes (pretty typical for very large data sets), incremental bandwidth usage past the initial storage is usually not much more than the data growth rate. As observed, it can be done in a way that respects privacy and safety.

        Cloud storage has two main advantages over local backup solutions. You won't run out of disk space, and it's off-site (so a house fire won't take out your data set). Any on-site solution automatically fails that level of redundancy. Storage on S3 is ridiculously inexpensive any more.

        I have about 6 TB of data that I need to keep backed up. I have about 12 years of digital photography and video originals, including stuff like wedding and honeymoon photos, as well as the birth and first years of my children's lives. When people suffer house fires, one of the most common and greatest laments are the things that can't be replaced - usually photographs.

        My solution is four tiers. I have a local RAID0 in my Mac Pro. I have Time Machine backups of that (this is hands-down the best consumer on-site backup solution on the market). I rsync those files to a local RAID10 NAS device (Synology are a bit pricy, but they are completely worth it, really excellent built-in software with a lot of features you might find surprisingly useful, and you can purchase expansion bays to extend capacity as you're running low). Then finally I back up to Amazon S3 in encrypted form with JungleDisk (I no longer recommend this software, I own a copy of it from before it was bought by RackSpace, the quality has gone down since RackSpace bought it and "improved" it, plus I gather you now have to pay a monthly subscription, AND pay for your own storage - crap).

        The only way my data is in jeopardy is if my house burns down (takes out 3 local redundancy & backup solutions) on the same day that Amazon has critical failure. And it's all 100% automated, Raid0 happens at time of write, TimeMachine alerts me if there's problems creating a backup and gives me local history, my NAS warns me by email & SMS if it so much as writes too slowly (my rsync cron script emails me if it can't reach the NAS for some reason), and JungleDisk does a nightly sync with S3, and sends me weekly reports so I can be sure that it's doing its job. I have quick local access, and slow offsite access if everything else fails (I'd probably go bum my work's huge pipe to do the initial restore if I had to rely on that).

        • by Nikademus (631739) * <renaud@@@allard...it> on Sunday March 25, 2012 @03:37PM (#39468589) Homepage

          "Storage on S3 is ridiculously inexpensive any more.
          I have about 6 TB of data that I need to keep backed up."

          So you mean that 6000/month*0.125$=750$/month is cheap?
          Or did I miss something?

        • Re: (Score:3, Insightful)

          by Anonymous Coward

          My solution is four tiers. I have a local RAID0 in my Mac Pro. ...

          You do understand what RAID0 is, right? RAID0 is strictly for performance and offers zero data redundancy or failure protection. In fact, since you need both disks to function to read your data - you're essentially halving the MTBF of using one disk. Perhaps you meant RAID1? (a mirrored set)

        • by BlackPignouf (1017012) on Sunday March 25, 2012 @04:07PM (#39468827)

          I have Time Machine backups of that (this is hands-down the best consumer on-site backup solution on the market).

          Did you actually use it for recovery?

          Both my rotated TimeMachines were corrupt. They never complained during backups, but failed miserably while trying to recover my Pictures HDD.
          Only some of the backup files were corrupt, but when you try to recover a complete disk with TM, it's all or nothing, and the process stops after the first error, leaving you in the dust.
          I had to write a parsing script with ruby, "cp -avX", ditto and chmod in order to get my system back.
          It wasn't so hard, but it sure was stressful with one disk down, two corrupt disks and no other backup to get my pictures back.

          BTW, TimeMachine doesn't backup every file in your system, and is too stupid to realize that it should not begin from scratch after recovery : it needs twice the storage after that, because it thinks every file is new.

          My drives weren't big enough, so I had to wipe the backups and lose the local history.

          Fuck it. I began using Carbon Copy Cloner since then, and never looked back.
          It's free as in donationware, it works, it gives you a bootable backup that you can actually test and rotate properly, it can easily be automated, it archives the files that you've deleted between backups, and uses much less space than TimeMachine.
          I hear SuperDuper is just as good.
          TimeMachine is some crappy software with nice looking interface that gives you a false sense of security.

        • by cheetah (9485) on Sunday March 25, 2012 @04:18PM (#39468913)

          S3 storage for 5TB isn't what I would call cheap. We are talking about $580/Month(or almost $7k per year). For that amount of money, you could buy a new set of 5TB worth of hard drives each month and then ship them to a remote location and pocket about $200 a month in savings.

          Not a perfect solution(no online access) but I think it underscores just how costly S3 still is for large amounts of data. If you are talking about a few hundred GB of data, S3 storage is cheaper(and better) than anything you could reasonably do yourself. But once you scale up the usage... Heck, you could buy and colo a remote server and ship drives back and forth for less than what S3 would cost...

    • by burisch_research (1095299) on Sunday March 25, 2012 @12:49PM (#39467407)

      You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.

      Cloud is very trendy right now, but that doesn't mean it's a one-size-fits-all.

      • by Trixter (9555)

        You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.

        This doesn't prevent the OP from using local backup in the meantime. I backup to local storage as well as cloud. The local backups complete quickly in case I need to retrieve a file, and the cloud is there for if my house burns down.

        The OP stated in his question that he has a lot of data but no money to buy redundant storage -- well, that's his real problem. If you have 3T of data, buy 3T of backup. I don't know what the OP is looking for other than a magic compression answer or something.

      • Re: (Score:2, Informative)

        by Anonymous Coward

        You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.

        Cloud is very trendy right now, but that doesn't mean it's a one-size-fits-all.

        Crashplan has an option where they will send you a hard drive to seed your backup locally and mail it back. That way you only have to do incremental backups once you do the initial seed.

        If there's no offsite backup, the whole scheme is worthless. What happens if there is a fire?

      • by Wrath0fb0b (302444) on Sunday March 25, 2012 @01:06PM (#39467555)

        You're assuming that it's encryption that's the problem. In my case, it's a problem with the size of data vs. how much bandwidth I can use. I get an allocation of 20GB a month, and even that's very expensive. Backing up my 5+ TB to the cloud is simply not an option.

        CrashPlan will let you Fedex them a hard drive to get the backup started. From then on, you only need to send deltas.

      • SparkleShare looks and works like Dropbox, but is actually just a fancy automated self-hosted GIT repo, (which you can interact with using GIT commands on a remote repo if that is what you want to do).

        The wiki explains how to encrypt things (and the encfs recipe doc'd on the wiki also works with Dropbox, etc.)

        I think the project has matured really well, but still isn't really well-known, and doesn't even get mentioned much on the slashdots, although that's where I heard about it.

        www.sparkleshare.org

        https:// [github.com]

    • by FridayBob (619244)

      "... there are cloud backup solutions that encrypt your data *before* it leaves the machine. ..."

      That's nice, but there are some other issues. First, a major one: he has too much data for that -- it seems to be measured in terabytes. Since he's complaining that blue-ray is too slow, his upload speed will likely make backups up to the cloud an impractical solution also.

      Second, there are some minor issues: cost and reliability. Even if he has plenty of upload bandwidth, having some cloud service hold on to that much data for you is not going to be cheaper than buying the extra hard drives yourself. Af

  • by wanderfowl (2534492) on Sunday March 25, 2012 @12:44PM (#39467357)

    One way to save a bit of cash is to buy a USB eSATA drive dock (single or double) with some bare eSATA drives. This cuts the enclosure out, and allows you to buy bare drives, which are often cheaper than enclosed drives.

    You could also consider Drobo or one of the Wiebetech multi-drive RAID containers. But encryption + cloud isn't all bad.

  • Budget (Score:5, Informative)

    by macemoneta (154740) on Sunday March 25, 2012 @12:45PM (#39467373) Homepage

    "large data collection and not a large budget"

    This is your problem right there. You can't enter into a a situation like this without planning a budget for the inevitable failures. I suggest purchasing a new larger drive (3TB are common now) and migrating the data from the problematic drive. Then migrate the data from several older smaller drives. This will reduce the component count (points of failure), save you power (cost in the long run) and keep you ahead of failures. You should plan on doing this periodically to maintain the integrity of the data.

    • And don't forget nightly incremental backups..
    • by AmiMoJo (196126)

      The old smaller drives could go in a dedicated NAS box using software RAID. Doesn't have to be on all the time, just for periodic backups.

      Look into MultiPAR for redundancy for your most important stuff too.

    • Yeah, in my view, there's a simple time-tested method for managing your data: Get a centralized storage medium big enough to hold it all, move everything to it, and then back that up. Your backup medium should be at least twice as big as your storage medium-- for example, if you have a 2TB drive that you're storing everything on, you should have a backup device that can hold at least 4TB.

      But it seems like he's saying, "I want to store everything and back it up, but I'm unwilling to pay for the storage med

  • Buddy NAS (Score:5, Interesting)

    by Anonymous Coward on Sunday March 25, 2012 @12:48PM (#39467405)

    I have a solution I call the "Buddy NAS". Go out and get two cheap computers. It could be a PC or a mini-NAS or a low-end server. Anything that will hold multiple hard drives. You jam both full of hard disks and use them as a backup/NAS server. One PC is kept at your place, the other at your friend's house.

    Both computers have an account for you and an account for your friend (it helps if your friend is nerdy and "gets" backup solutions). Both of you now have a backup solution in your own home and a remote backup server at a friend's place. Two copies of your data, one remote. Basically it's like having local and cloud storage for you and your friend and it'll cost less than a grand if you shop around. If neither of you have static IPs you can use dyndns.org to connect to the remote boxes. Bandwidth shouldn't be an issue if you use rsync to backup changed files nightly.

    • by toygeek (473120)

      I wish I could mod that up, its actually a pretty decent solution with lots of potential for win.

    • by Teun (17872)
      An excellent solution for those with enough bandwidth.

      I see one little problem, these computers are Always On' and will fail sooner than a disk that's connected once a day or once a week.

      That's why I have two large HD's, one at home and one at the GF's place, we rotated them about once a moth.

  • I have invested in USB backup drives of about the same total capacity as my primary storage drives. Yes, that's a lot of hard drive space for backups, but it's really the only practical solution that I have found. Just think of it as the cost of not losing all that data to the inevitable drive failure.

    An external eSATA drive dock and a stack of 2TB drives might be a somewhat better way to go about it, at least backups and restores would be faster than the USB drives.

  • by jchawk (127686) on Sunday March 25, 2012 @12:52PM (#39467433) Homepage Journal

    I can offer a couple of suggestions... What I did was buy a used Dell Poweredge 2950 on eBay for about $500 bucks shipped and I added 4 x 1tb SATA drives to it and I run a raid 5 setup with 3tb of usable space across the four 1tb drives. This solution cost me less then $1000 and I have a nice playground to experiment with VMWare ESXi.

    I know that's not exactly budget conscience but it works great for me.

    If I were on a tight budget I would just buy a 2tb USB drive from Newegg or somewhere similar. It looks like you can buy a name brand for about $130 bucks.

    If you have a little bit more money to spend you could always buy a couple of 2tb internal SATA drives and run RAID-1 mirroring on them. You could put these into an old computer and make a little NAS linux server...

    If you're saying you have no money to spend then maybe you need to consider cleaning up your data. Often times all those "personal files" that you think you need to keep... Really aren't required. Just my 2 cents but this problem is very solvable.

  • I delete stuff (Score:5, Insightful)

    by Amiga500_Rulez (988955) on Sunday March 25, 2012 @12:53PM (#39467441)
    Seriously. How much crap do you really need to keep around?
    • by petes_PoV (912422) on Sunday March 25, 2012 @01:05PM (#39467549)
      Deleting stuff is all very well. But unless you just do an "rm -rf *" and just be done with it. you need to invest some time in deciding what to remove, what to keep and whether that directory called family-photos really does contain what you expect it to. Even at minimum wage rates, the time spent trawling through a couple of TB of "stuff" could easily exceed the cost of a new disc - and then a background copy / backup onto it.

      Obviously you still have an issue of tracking things down on the rare occasions when you actually need some of your family photos. But you can rest assured that they're in there somewhere and weren't purged last time you needed a few GB for more webserver logs.

      Maybe the first step is to de-dup the existing data. You'll still have some manual intervention to check possible duplicates, but it's a first step towards tackling the bigger problem.

      • by swalve (1980968)
        Agree. It is way easier to just buy new drives and schedule any purging you need to do for some later time. That way you can decide what stuff is important without the cloud of impending failure hanging over your head.
  • Cheapo used market PC, invest in some large drives and a couple of drive docks, install FreeNAS.

    Take a weekend to organize your data however it makes sense (by year, subject, file type, whatever), and store it on a particular drive. Rinse wash and repeat. Depending on how important the data is, store in a fireproof safe onsite, or offsite. When (read: if) you need the data again, dock the drive and retrieve.

    Personally, I'm about to liberate myself from years of data. I'm tired of all these bloody drives

  • One, you can't have enough backup images as something always seems to go wrong. Should include at least one offline unplugged "safe" unit. I know, it's a hassle to keep them up to date.
    Two, the longer you wait the less all this backup space costs, so don't buy too much too soon.

  • Magic (Score:5, Insightful)

    by lucm (889690) on Sunday March 25, 2012 @01:00PM (#39467499)

    So your disks are full and possibly broken. You don't want to have more disks, you don't want tape or optical medias, and a storage provider (aka The Cloud) is not an option... Then you have three solutions "down the road":

    1) Delete stuff
    2) Invent a new compression algorithm that will allow you to reuse the same disks forever without losing data
    3) Rely on magic*

    *might overlap with solution #2

    • Close ...

      1. data deduplication. Most people have a lot of redundancy ever since hard drives started growing past a few gigabytes.
      2. for photos - people might have 20, 30 shots, of which only 2 or 3 are really "keepers." Get rid of the crapola. Same goes for videos that are "better off forgotten" or that, while still viewable, wuld be a pain to watch because we no longer can stand looking at 320x200.
      3. tar lots of little files into one big file, so as to not lose disk space to 4k sectors holding a 3

  • It depends on the data, but many formats compress really well when using WinRAR. Many of my files, for example, that reach nearly 10:1 compression. Unless we are in the same profession, I wouldn't set your expectations that high, but I imagine on average you could get your data usage down to 40%. If I'm right, maybe you could winRAR several folders from the failing drive to the smaller drives, and not necessarily need to get more space available.

    That said, I really do think the suggestion of buying an

    • I went the compression route myself. It was effective. Then it became more than effective: It became fun!

      Now I spend many an evening studying and devising new means of compression by which I may squeeze a tiny bit more useful information into those bits. I'm good at it, too.

      Also... in my very extensive testing experience, though winrar beats zip with ease, the contest is close between rar and 7z on small files - and once the files get big, the advantage of 7zs -md=256m dictionary size gives it a clear edg
  • As several people have said you already answered the question yourself. Spare HDD + Blueray.

    You can achieve what you want by also changing the way you think about your data.

    How much of your personal data is live? As in, how much of it do you access constantly, and need immediate access to?

    Here's what I do, I have discrete HDD set up for each data type (not needed...but I had spare ~500gb drives so it's how I did it) There are broken down to Music, Projects, Video, and Photos. Each of them is synced mon

  • There is a solution for the problem of "too much data": it's called "retention".
    You must give up some of it, or transfer it to some other, long-lived medium.

    Otherwhise, I suggest you face reality and invest accordingly [youtube.com]

  • by mveloso (325617) on Sunday March 25, 2012 @01:02PM (#39467519)

    Drobo -> mostly reliable local backup
    BackBlaze -> mostly reliable offsite backup

    You might want to substitute a ZFS-based FreeNAS for the Drobo, if you're so inclined. It's less automatic, but seems just as reliable.

  • All of my personal data is in my home directory and easily backed up to non-volatile media (which I do a few times a year, but not as often as I should.)

    All of the project data is on SourceForge or company project servers, so there are duplicate copies of that.

    I hardly think of my music or movies as "personal" data nor as irreplaceable. Were I still playing video games, I don't think I'd bother backing up game data, either.

    When people talk about needing entire drives for their personal data backups,

  • Just delete it! (Score:2, Insightful)

    by Lazy Jones (8403)
    If there's some personal data you're missing at some point, just ask Google or the NSA ... But seriously, I've never made backups and not even bothered to copy over stuff from old PCs to newer ones when I upgraded (I keep old hard drives in a closet just in case there's something old I'm missing, but I never really do). The only personal stuff I keep safe is images on my iPhone (backed up on the PC) and email (safe-ish on the server at work). If I needed more space, I'd go with Wuala [wuala.com] due to its relative saf
  • My fileserver uses RAID and makes a separate (encrypted) backup to an external USB hard drive (fortunately, my data hasn't grown faster than hard drive sizes so I can fit it all on a single 2TB drive, to ensure file integrity, periodically I have rsync verify file checksums,)

    As a secondary backup, I use a 1TB notebook drive locked in a USB enabled fire safe:

    http://www.amazon.com/SentrySafe-QA0121-Fire-Safe-Waterproof-Storage/dp/B00166187Q/ref=cm_cr_pr_product_top [amazon.com]

    I used metal straps to tie it down and lock

  • by MarcQuadra (129430) on Sunday March 25, 2012 @01:07PM (#39467581)

    I think it's time to admit that you're a hoarder. What exactly -is- your personal data that's so precious? I run a server just to keep my skill set up and run my side business, but I've only managed to accumulate around 600GB of data, only about 35GB of it is 'mine', the rest is client backups.

    So first admit that you're a hoarder, then decide if you wan to address that issue or indulge it. If you choose to indulge it, you're going to want to build a small home server. Something with a low-end 64-bit CPU (i3?), a gigabit LAN port, and lots SATA ports and 3.5" drive bays. Buy a bunch of high-quality (WD RE4?) matching drives that fit your data needs times two (you're RAIDing space away). Once you have that, install Linux on it, build a software RAID-1 or 0+1 array (don't do RAID-5 unless you can handle days of rebuild time), and format it with something accessible (read: in the kernel, like EXT4). Create a share on the array with Samba and happily access it from all your machines (don't bother with Netatalk or NFS; CIFS is great on all platforms). As your data needs grow, you can add drives in pairs or replace drives with larger ones and grow the volume. If you need backup, you'll want another array, preferably on another low-end box (an enclosure on your desktop?) but it can be built on a RAID-0 or JBOD to save money.

  • by QuasiSteve (2042606) on Sunday March 25, 2012 @01:10PM (#39467599)

    It's that time again, is it?
    http://ask.slashdot.org/comments.pl?sid=2452630&cid=37557630 [slashdot.org]

    Either..
    A: Buy that HDD. Yes, they're a bit more expensive right now ..or..
    B: Wait a few months, prices will come down again, buy that HDD then. Yes, you may lose your data in the mean time.

    Now stop asking or I'm going to pull over.

  • Drives die, sometimes without warning (and old statistic by IBM says 50% of the time there is no warning). You could just throw everything away, as you are going to lose it anyways, sooner or later. Or you could find the resources needed to make sure you have everything on at the very least two drives (one of which should not be connected or running). There is nothing that can replace reasonable backup.

    As a side note: Common sysadmin wisdom is to have 3 independent backups in addition to the original.

  • by joocemann (1273720) on Sunday March 25, 2012 @01:10PM (#39467609)

    You might be on the next spinoff of Hoarders programs, a digital hoarders show.

    In this show, redundancy, old versions, and files that haven't been opened in 5+ years are brought into question, for which you will be embarrased to defend... You will attempt to justify why you still have linksys drivers for a wrt54g you don't even have anymore. And no, the DVD ISO of the Alvin and the Chipmunks movie, that you never burned or watched, is not worth saving.... Neither are about 85% of the digital pictures you took (you know, the ones that were the 'bad shot' that you took before finally getting the good one).

    Take a day or two, go through it chunk by chunk, and purge! PURGE!

  • A friend and I each have FreeNas servers with multi TB raid-z.

    Some of our data we keep mirrored between them.

    The servers are physically brought together occasionally for a full sync, but most of the time rsync -n is done over the internet to see what needs to be updated and the data transferred on a removable hard drive.

  • I just put everything on Facebook -- hey, where's everybody going?
  • Answer:

    Floppies

    More floppies

    More floppies

    (I can sell you some cheap!)

  • As others have noted, if you can't afford AT LEAST another drive, serious problem right off the bat. One wonders what the data is worth given this.

    I'll move on, assume the data is worth AT LEAST another drive or two (we're talking a couple hundred bucks at most, come on):

    1. drbd: raid to a low cost, remote machine with similar sized drive. Dead drive is now recoverable.

    2. amanda or similar backup to drive on remote machine. No, not tape, just virtual on disk. Now have a backup history as well in case one ne

  • Get older version LTO drive, get identical harddisks with spares and do full drive image backups with incremental backups in-between.
    Mirroring disks (Raid 0+1) is good too.

    All that other stuff - DVD/Blue Ray, cloud is chickenshit. Drives die or act up.

    Run Linux on one drive and do image backups from there.

    Zerofill the empty space,
    dd if=/dev/zero | split --verbose -b 2000m -d - ZERO

    compress the image:
    dd if=$DEVICE | gzip -v | split --verbose -b 2000m -d - NAME

    and write the chunks to tape.

  • by BoRegardless (721219) on Sunday March 25, 2012 @01:28PM (#39467763)

    People have different needs. Some needs are imposed by either employers or the wonderful US Govt. for mandatory data retention. Others are your life's design work that you want to retain until you die. Other data you want to pass to your kids. If you can't afford to lose it keep multiple backups on multiple media in multiple locations. Books & pamphlets have been written on this. Transfer the data to new media once a year or two or three & keep all working drives.

    No single storage device local or remote is immune from disaster. The Alexandria Library succumbed and took with it countless early human treasures. Wars have done in archives all over the world. Lightning, outages and power surges can defeat the best protections even when electronic equipment is turned off, but still plugged in (laptops are better when left unplugged, which is actually a great asset).

    Backup is one thing; recovery is another and it can be GUT WRENCHING. The recovery process needs as much thought as backup.

    A Clue or Two: A business partner had his MBPro backed up to 2 external HDs. Not great, but OK. Said MBPro crashed on the Lion upgrade. No way to know whether it was hardware or software and the MBPro should have at that point been off limits for use until carefully checked out. He happens to live in an area subject to lightning and outages which can affect anyone (even with a UPS). However, he reinstalled the Snow Leopard and plugged the first BU HD in an attempt to reload the data; HD became corrupted. Should have stopped, but then the 2nd HD was corrupted. Moral of the story; Recover data from a backup to an external HD running on another computer than the one that got mucked up.

    The cost of 3-4 external 2-3 Terabyte hard drives and a couple cases or RAID box is dirt cheap compared to the value of the hours you put in on your computer each year as are Blue Ray drives & disks.

    Caution: Someone on this list mentioned putting drives and disks in a "fireproof safe" or "fireproof file cabinet"; wrong! The UL approved boxes are designed only to protect "paper" for a given amount of time in a typical fire by releasing steam (212 deg. F = goodbye DVD/BR disks). Once the fireproof agent uses up its water...Farenheight 456 takes care of all contents...permanently. This is why multiple locations are needed.

  • I'm assuming to start with that you have backups of everything in some fashion, with which you could put it all back together if your biggest drive suddenly failed spectacularly.

    In that case, how important is uptime to you? Since this is personal data, I'm guessing that you could live without live access to it for a few days. And given that, I think your best bet is not to keep a spare just sitting around, but to only buy one when you need it. Hard disk prices keep going down, and the price for the sa
  • SDLT tape drive and some tapes. If your "personal data" is not worth the $800.00 to buy a good used SDLT drive and a few tapes, then it's not worth backing up.

    Just do not dink around with theoretical "backup solutions" that are not proven. and no, hard drives are not "proven" for reliable and long term backup. I have DLT tapes from 16 years ago that I know for a fact I can still read.

    If your data is important, You dont screw around with consumer hard drives that are known to have a low MTBF.

  • My personal and family data (not including ripped DVDs etc) are about 1 TB. Mostly photographs and video with my DSLR so the files tend to get large...but I also have a ton of documents, app installs, and all sorts of misc data. I must admit I'd be curious as to what fills multiple external HDs for personal data but to each their own.

    Good organization outweighs medium in my case. 2xExternal 2 TB HDs - primary and secondary...and then a third stored off site at my parents that I update about 3 times a year, so if the worst happens I'm 6 mons out of date, but its usually about 4. And thats if both my primary and secondary go down. Thats a cost of about $300 total and a little a bit of effort.

    "A little bit of effort" is defined by how you organize. Backing up manually means I don't rely on software or a service, but it requires some forethought. For me I break it up by data type and usually year...sometimes I go one more by how that data was acquired (photos I add who took the picture). This is important because I put anything new into a diff folder so I know whats new and whats not. It took me a couple of years to get to the structure I have but I sometimes add small tweaks. The effort or time now is fairly miniscule.

    What I'm trying to get at is this : if you're prepared to put a small amount of time in every now and again, with an initial overhead, you can do this very easily and cheaply.
    • I should add that with multiple TBs of personal data you can't pick something that is 100% fail safe, cheap, and effortless. You can only pick 2 of those. 40 Blu-ray disks equates to a 2 TB external HD. The odds are one of those Blu-rays will fail. I work off the principle that as soon as one of my externals start to cause problems I go out and buy a new one and remove the old one out of service - its not worth the trouble. Last time that happened was about 2 years ago.
  • by jon3k (691256) on Sunday March 25, 2012 @02:16PM (#39468075)
    You ruled out: hard drives, tape, discs and cloud storage. What exactly do you expect us to say here? There isn't some other magical form of storing data we've been hiding from you.
  • by IndustrialComplex (975015) on Sunday March 25, 2012 @02:19PM (#39468087)

    First, let's look at your problem: You are gathering too much data. Either the data is 100% needed and irreplaceable, or it isn't. If it isn't, your first step is to treat your data just like you would physical junk that accumulates in your house.

    Create Three folders.
    1. Critical Keep
    2. Unsure
    3. Toss

    Go through your data and MOVE it to one of those three folders. If it isn't critically important data that you would be upset that you lost and can't be recreated (wedding videos, etc) It goes in the Critical Keep folder. If you aren't sure about it right now, but you can't declare it for folder 1, put it in 2. Anything else "old install files, backup data from a windows 98 machine, etc" That stuff can be deleted. Be harsh with yourself. Think of it like moving from house to house, if you haven't opened that box by your third move, just toss it in folder 3.

    Repeat the process until you either have everything in your Critical Keep folder, or your delete folder.

    Now, hopefully you have reduced the size of the data you are using to something marginally manageable. I'm a data hoarder, and I've managed to keep the rate of growth of my data to lag behind the general rate of growth of HDD capacity. Now for the fun stuff:

    Two things you want to avoid.

    1. Loss due to a dying disk
    2. Loss due to a destroyed home (fire, theft, etc)

    Here was my budget solution that resulted in a fire and forget backup system that is suitable for a home user and is about as minimal as you can get for cost.

    3 Disk Drives.

    A primary drive to run the operating system and hold installed programs and two LARGE data drives in a RAID1 configuration.

    Static data files (Video, pictures, etc) get stored on the RAID1

    A scheduled process (once per month for me) backs up the OS drive to a virtual HD file on the RAID1. The files on the RAID1 are then backed up to a cloud storage service (Carbonite in my case).

    So, what is the result of this?

    My operating environment is backed up monthly. The only thing I lose here is configuration changes or programs installed since the last backup (less than 30 days for me)

    The RAID1 ensures that my personal/static data is protected from a single disk failure, and helps a bit with read performance for the static (and large) files.

    Should a cataclysmic failure occur and my entire computer is lost to something like a fire, remember that I've been sending what is on the RAID0 out to the cloud (carbonite), so when I can rebuild a computer I can just download the (very large) offsite backup from the cloud to my new machine.

    The downsides I have right now:
    1. I maintain the windows backup as a VHD file because it allows me to ensure that the backup data is 'packaged'. I don't know the exact details about windows backup, but given that Carbonite sometimes excludes system files I didn't want to risk an important hidden/system file being missed in the backup. In addition I didn't like how it could only backup to the root folder of a drive. The downside is that the resulting 100GB file is a pain to backup, which is why I restrict the backup histerisis to 30days (previously I had it backup every 3 days) This keeps it from continually uploading the VHD file to carbonite.

    2. The HDDs for the raid1 lose half their total capacity in that configuration. I used it because it let me only have to use 2 drives and the performance boost. If you can afford 3 drives, go for a RAID5.

    3. Most Motherboards support RAID natively now. However, I understand that you can run into issues with hardware RAID if you have to switch to a different hardware solution. I haven't tested this, but it could potentially be an issue if you use a RAID5 from hardware and your motherboard fails and you can't replace it with an exact model. The good news here though, is if you have been backing up to the cloud, typically it's done on a per file basis, and thus you don't have to worry about this. Just download your stuff ba

  • by denmarkw00t (892627) on Sunday March 25, 2012 @02:41PM (#39468223) Homepage Journal

    Delete your porn

    The rest of your personal data will fit on a floppy.

  • by BoRegardless (721219) on Sunday March 25, 2012 @02:44PM (#39468255)

    The words "Fireproof Safe" is by the definition of Underwriter Labs. It merely refers to being fire resistive for a given amount of hours in a typical fire "for paper".

    Forget DVDs and CDs and any hard drives surviving a fire in one of these "fireproof" devices. They are designed to release steam to keep the temperature at 212 def F until the fireproofing material exhausts all its water at which time the temperature goes up and, well...you can imagine what.

  • by Janek Kozicki (722688) on Sunday March 25, 2012 @02:46PM (#39468263) Journal

    My solution to this problem is painfully simple: about 5 years ago I bought 5 drives 500GB each. I have put a server (made from old parts, like pentium IV and so on) in the basement (where nobody hears it, and it can be as noisy as hell). I installed debian on it and configured cron to call rsnapshot three times per day for doing automatic backups of all PCs in my family. I never touched this machine since then.

    With one exception: 3 years ago I started to run out of space, so I bought 2 HDDs 2 TB each, reconfigured raid6, which was extremely easy because for raid I am using mdadm, which supports such operations online. Also I had few more spare drives during the years, so I kept adding them to the array, and currently there are 9 HDDs in this PC. It is very noisy, but nobody cares about that.

    It runs flawlessy, untouched for years, and nobody cares about it, except for when somebody in my family accidentally loses or deletes a file. Then suddenly backup comes very handy.

    Rsnapshot is especially good, because it keeps hardlinked copies of data from last week, 2 weeks ago, last month, and much more, depending on how you configure /etc/rsnapshot.conf. Currently I have backups dating back about 2 years, with granularity of 1 month. And it only occupies the space on HDD to reflect the changes between data, thanks to hardlinks.

    So my raid6 array has total size about 4TB and still 500GB free. And I feel this will last at least a year or two. In case of problems I can start deleting copies that are more than 1 year old. While most recent snapshot uses about 2 TB or such.

    Rsnapshot also can backup windows machines, so you don't need to worry about compatibility. Though I don't have windows machines and I don't test that in practice ;)

  • by professorguy (1108737) on Sunday March 25, 2012 @04:21PM (#39468953)
    So they've convinced us that WASTEPAPER BASKETS must be plugged in at all times (shredders and so-called "electric dustpans"). And I see everyone out with gas-powered BROOMS. And even the SAP which drips freely from our maples is, in modern sugar houses, vacuum pumped to a tank.

    And now there's 200 comments where the people are proud of their kilowatt server arrays which are powered 24 hours a day for their PHOTO ALBUMS? Are you people shitting me? I mean, you're putting me on, right? You don't really use up 10,000 kWH per year storing your family photos, do you?

    Hey, I've just invented the electric elevator-button-pusher. I save a TON of finger wear and tear.

    Sometimes, humanity makes me sad.

  • by kikito (971480) on Sunday March 25, 2012 @04:37PM (#39469095) Homepage

    Not trying to troll here. I'm serious. Consider that you might be simply storing too much stuff.

  • by multimediavt (965608) on Sunday March 25, 2012 @05:16PM (#39469387)
    I am making my way through the comments, and want to clarify a little. I am not talking about backup. I am asking about disaster recovery or just plain drive maintenance tasks that should be done annually. The drives are my backup. Yes, good corporate data storage practice is to have spare drives around. I am talking about home. How many have 2 TB drives sitting empty on a shelf at home, just in case? I don't know anyone, personally, and I know hundreds of geek admin types of all ages and experience levels, myself included. We usually buy storage upgrades as needed and seldom have current technology, large drives just laying around because we're using them! Other than that, great stuff so far. Thanks all.
  • Data integrity (Score:4, Interesting)

    by thereitis (2355426) on Monday March 26, 2012 @12:35AM (#39471881) Journal

    This is my personal data and it is starting to become unbelievably unruly to deal with as far as data integrity and security are concerned.

    Keep all your important files in a version control system. Personally, I use Perforce (it's free for 2 users or less). That gives you: multi-revision history and checkin comments, an easy way to pull a subset of files to any computer in your house, and peace of mind that you don't need to worry about kids deleting anything important as it's all stored on the server with history. Also easy to see what has changed on any computer and check those files in. And there's a big win for data integrity checks: Perforce stores the checksum of all files (and revisions) and can easily check that every file still matches the checksum in the central database. If you have any disk corruption, you'll know about it when you run 'p4 verify -q //...'. You can store files of several gigabytes each with no problem.

    On top of this, I use rsync to copy the server data onto backup drives. I'm also looking at storing backups online, but haven't taken that step yet.

    I've been using this system for years and I couldn't imagine being without it. It's so easy to find and retrieve exactly what I want - my resume 5 revisions ago, my tax return, photos from 2003. Even without that, the data integrity checks give a lot of peace of mind.

Are you having fun yet?

Working...