Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Data Storage Graphics Software

A Storage Solution for Lots of Digital Photos? 122

Posted by Cliff from the photo-album-equivalent-for-your-megapixels dept.
Duizendstra asks: "I've been asked to explore the digital storage possibilities for a professional photographer. One of the characteristics is the rapid growth of the amount, and size of pictures. At the moment, one photo session produces about 2 GB of raw data. He has an Apple - Power Mac G5, and he currently uses DVD as his storage medium. However, he has lost quite a few photos because of DVDs that can't be read anymore. I would like to know if any Slashdot readers have any experience in creating a solution for such a problem? Any help/idea(s) would be greatly appreciated!"
This discussion has been archived. No new comments can be posted.

A Storage Solution for Lots of Digital Photos? More

A Storage Solution for Lots of Digital Photos?

Comments Filter:
  • Get an extra hard drive and use it. Hard drives take a lot to kill... at least for me they do. ANyway, do something like that. SHould work. Eventually you'll run out of room, but you can always swap out/add another one

    • Re:Extra Disk (Score:3, Interesting)

      by HughsOnFirst ( 174255 )
      I'm a photographer and I'm looking at over a terabyte of external firewire drives piled up on my desk and spindles of DVDs backing them up. Right now the options for reliable long term reasonably prices storage pretty much suck if you generate around six gig of files a day. Lots of hard drives is fine as far as price goes, but they aren't an answer for long term storage. If anyone has an idea for storage in the 30 year range, I'd like to hear about it. My experience with tape back in the DC250 days was pret
      • ...but I'd love a recommendation for a system that I could trust for more than 10 years.

        Magneto-optical disks. Usually they should keep your data for around 50 years. They're very reliable, but have two cons:

        - They're expensive (around $17 for a 2.3GB disk)
        - You'll need 3 disks/day if you generated 6GB of data daily, so after a few weeks you'll need lots of space to store them.

        • Magneto-optical disks. Usually they should keep your data for around 50 years.

          The MO backups from the Sun workstations where I work have been found, through bitter experience, to have about a 5% failure rate.

          They suck shit.

          Hard disks and more hard disks are the way to go.

          -ccm


      • Lots of hard drives is fine as far as price goes, but they aren't an answer for long term storage. If anyone has an idea for storage in the 30 year range, I'd like to hear about it.

        No one does, because, frankly, digital photography hasn't been around that long--so there are no solutions that were around in the 70s, that are now still working, to demonstrate that they are reliable over that kind of time period.

        I think stacks of hard drives would suck, and could still fail on the shelf. I, like others abo

        • You (plural) don't understand. I and MANY others are looking for a reliable long term OFF LINE storage. Hard disks probably are fine if you are only talking about the platters and not the moter/bearings/heads/electronics not to mention the interface to the computer.

          Raid sounds good, but is there a standard for the controlers? Software to split the data across several drives in a similar fasion to raid 5 might work but I'd only be interested in it if it was open source in a language that was likely to be ar

          • Raid sounds good, but is there a standard for the controlers? Software to split the data across several drives in a similar fasion to raid 5 might work but I'd only be interested in it if it was open source in a language that was likely to be around for a while (Perl ?)

            Yes, there is a well established standard for the controllers. RAID is an enterprise-level technology that has been around for atleast a decade and is the storage medium preference for corporations that literally can't afford data loss an

          • I have some paper tape from 1972 that is probably readable

            And that, sadly enough, is your best solution.

            A few years ago I was brought in as a technical consultant for a city records office that wanted to modernize their records : a fireproof building full of flat metal storage racks full of deeds, records, and drawings (think civil engineering dating all the way back to the 1600's.) They needed to insure that whatever direction they went in modernizing the office would be viable not 5, 10, or even 50 years

          • Re:Extra Disk (Score:3, Informative)

            by WhyCause ( 179039 )
            I and MANY others are looking for a reliable long term OFF LINE storage.

            Well, (and this is me just talking out of my ass here), you could maybe invest in a film recorder [medgraphix.com] (we called it the slide-shooter). Think of it as a digital projector that projects onto film for later development. We used to use one in my lab to tranfer presentations from PowerPoint to slides (for scientific conference presentations), but I imagine that, as a last resort backup solution, it might work well for photos. The slides ar

        • No one does, because, frankly, digital photography hasn't been around that long--so there are no solutions that were around in the 70s, that are now still working, to demonstrate that they are reliable over that kind of time period.

          I'm unsure if you think this is a digital photography problem, or if you just believe digital photography is the only possible reason someone would need massive, long-term, reliable storage.

          Either way, it's an "ask slashdot" at least twice a year, for all sorts of reasons. It's a

      • Re:Extra Disk (Score:3, Informative)

        by Nutria ( 679911 )
        I don't ever see any tape systems touted for long term storage, but I'd love a recommendation for a system that I could trust for more than 10 years.

        SuperDLT. It's what we use for SARBOX data retention compliance.

        Unfortunately, that's "enterprise" tech, which means Big Bucks.

      • Re:Extra Disk (Score:3, Informative)

        by tonsofpcs ( 687961 )
        Film. No, seriously, there are labs that will project your digital files onto slide film. Usually you can find them if you look for presentation preperation companies. They usually advertise this service for converting computerized (read: PowerPoint) presentations to slides for showing in venues that do not have video or (SX-/X-/S-)VGA projectors. Good film has proven that it lasts for a long time. Go check your 30 and 40 year old slides.
      • 30 year range sure a couple of Ideas
        1 best, home-made platinum prints, gives the best combination of storage volume and longevity
        2. staying digital, tape system as in punched mylar longest data retention, but storage volume has major suckage factor

        what I would do is massive RAID 50 array in the back for long term archial storage in the back room, something not as massive and faster up front for recent work that you might actually need to get to; only fire up the back-room arrary if you need something.

        I know
    • I too think external hard drives are the way to go. Frankly, I haven't had a hard drive fail on me in the last decade, so I think it's unlikely that the computer's hard drive and the drive that stores the backup will go at the same time or in a succession that's too quick to get replacements before the remaining drive goes. If you are that afraid, get a second backup drive. If your data really is valuable, I bet that the extra hard drives are cheaper than replacing the data. At any rate, the external sh
  • I'd consider setting up a file share (nfs, samba, etc.) that is on a machine with plenty of storage space. ReiserFS 4 offers a compression module that would be perfect for this. It would make sense to set the FS to compress heavily, since transfer speed is less critical. Your friend could probably get about 6:1 compression I'd guess, which with 400 GB of storage in an LVM array would hold over 1200 photo sessions. This should be enough to last him a few years. In all liklihood in 3 years he'll be able

    • Re:Reiser4 with compression (Score:5, Insightful)

      by swillden ( 191260 ) <shawn-ds@willden.org> on Saturday November 26, 2005 @01:54AM (#14117245) Journal

      Your friend could probably get about 6:1 compression I'd guess

      I *highly* doubt that. It's unlikely he'd get any significant compression, and very possible that compressing the files would actually increase their size.

      We're talking about photos here, which are already compressed. Even RAW photos are compressed heavily (though losslessly). For example, a Canon EOS-1Ds Mk II takes RAW photos at a resolution of 4992 x 3328 with 36 bit per pixel. An uncompressed image would be 4992 x 3328 x 36 / 8 bytes, which is about 71MiB. The image files produced by the camera, however, are 14.6MiB, a compression ratio of nearly 5:1. The file system compression isn't going to get much more. On RAW files from my camera, bzip2 -9 only averages about 0.1% reduction in file size, and bzip2 -9 is very good -- and very slow -- compression.

      Disk drives are the best way to safely store large volumes of data, especially when you add some redundancy, but don't expect to get any help from compression of already-compressed data.

      • That's not quite correct. Remember that RAW files contain the sensor Bayer pattern data, not actual pixels. The conversion to pixels happens during the demosaicing process which, in the case of RAW files, happens later on in your computer, not in the camera. In the case of the 1Ds MkII, the camera has 4992 x 3328, each of which is 16 bits (actually 12, but I think the files are stored are written as two bytes/photosite), so that's about 33 MB.

        Nikon DLSRs give you the choice of storing RAW files as either co

  • Best bet... (Score:3, Informative)

    by tenverras ( 855530 ) on Saturday November 26, 2005 @12:52AM (#14116935)
    Would be to buy a few external hard drives. With the storage capacity of hard drives these days you can't go wrong. I bought an enclosure for one of my internal drives and now I don't know how I could live without it. Having a portable drive like this is an amazing convience, especially with a capacity of 160GB.
    • External HDs are the way to go.
      Plus, I know that the apple shop sells a 1 TB External if needed.
      Idealy you keep the pictures you are working with localy and back up to the external. They are very easy to use, even for those who aren't up to par with technology.

      • Re:Best bet... (Score:3, Insightful)

        by jrockway ( 229604 ) *
        No! These LaCie drives are a bad idea. They put multiple disks in one enclosure, but they're not RAIDed. That means if one disk fails, your entire life's work is gone!

        I would store these photos on a dedicated server that has good RAID. I don't think full tape backups are an option, but remember that RAID will handle a single drive failure -- not you accidentally typing rm -rf * . Maybe there's some service that will mirror your important data off site somewhere. That's probably expensive, but if this
        • I've noticed that most companies specializing in offsite backups charge 5-10x what it'd cost (apart from one-time labor) to set it up yourself, and do it right.

  • perhaps the problem is with the DVDs? (Score:3, Insightful)

    by ubiquitin ( 28396 ) * on Saturday November 26, 2005 @01:08AM (#14117026) Homepage Journal
    One approach is to burn three copies and then you can recover the data by averaging the signal between them. This requires multiple drives but it is better than having to give up on archived data. Manufacturers suggest: "Store your recordable DVDs vertically, protected from sunlight, in a room that avoids wide variability in temperature and humidity."
    • I hear stories about people not being able to read CDs/DVDs that they've burned, but I have to wonder what they're doing wrong. Whenever I burn, I verify the media to ensure that what I burned actually matches what was on the drive. I've had some problems with crappy drives (I'll never buy another NEC burner again) and some bad media (which I immediately destroy), but once I've verified them then I never have any problems reading them. I always burn 2 copies - one for the safe at home (for fire protectio

      • Re:perhaps the problem is with the DVDs? (Score:5, Interesting)

        by Calmiche ( 531074 ) on Saturday November 26, 2005 @01:59AM (#14117267)
        My problem is that I'm finally starting to see decaying CD's. I've got some backup CD's from 1998 or 1999. (I can't access the media anymore to check dates, but somewhere in there.) I put them in the other day to look for some old data and they wouldn't read. When I pulled them out of the drive, the silver media was peeling away from the disk. I've run across about 6 of my backup cd's so far this year that are doing the same thing.

        No, I think a good harddrive array is going to be your best bet. Get several harddrives and mirror the data. The cost of gigabytes is dropping on a daily basis. You should find that when you need more room, it will be easly upgradable and cheaper as the years go on.
      • Even verifying CDs after burning isn't enough. I've had one batch of CDs that I burned stuff onto and verified that the sha1sums of the files I had burned on them matched. ONE week later the CDs became unreadable (I got a whole lot of I/O errors trying to even read the CDs). These CDs weren't even in the sun or hot car or anything. And these CDs weren't scratched and didn't have a spec of dust on them that I could see. They were in a cool desk drawer and untouched for that week and lost their data in a
        • This won't necessarily work for you, but I've sometimes found that if some files can't be read from the CD-R media itself, that making an ISO of it on your hard drive, and mounting that using Daemon tools or something allows the individual files to be read correctly. I have recovered data from 2 old CD-Rs this way.
      • I just wanted to mention that a fire-proof safe isn't necessarily going to protect your CDs from a fire. The operating principle of most fire-proof safes is that the safe is vacuum-tight and very thick. So there's no way for the stuff inside to catch fire, however it does get hot, so hot that, if you open the safe while it's hot, the contents will burst into flame. So if it gets that hot, you may find your CDs melted, or at least damaged, even if they don't exactly burn. Your paper documents should be f
        • Yeah, I realized when I bought it that it wasn't going to be perfect. The idea is to get maybe 75% of the way there, and if I really truly lose the media then I'll revert to the safe box at the bank. The same can happen at the bank too - a fire so hot that most stuff melts. Thankfully media can be replicated easily and kept in two places.
        • Your paper documents should be fine though.

          Well, that is unless the temperature inside the safe climbed so high as to make the CDs melt to the paper documents.
      • Whenever I burn, I verify the media to ensure that what I burned actually matches what was on the drive.

        One problem with that is that CDs and DVDs use error-correcting codes. Unless you know how many errors have been corrected by the drive, you can't tell the difference between a perfect disc and one which is just below the threshold of having uncorrectable errors.

  • Serious OS X user? (Score:4, Informative)

    by presearch ( 214913 ) on Saturday November 26, 2005 @01:10AM (#14117033)
    He should get an Xserve RAID, of course.
    It'll just work, it's well integrated with his G5, and it's cost effective.
    • ...and Aperture for image management.

    • Re:Serious OS X user? (Score:4, Insightful)

      by karnal ( 22275 ) on Saturday November 26, 2005 @01:59AM (#14117269)
      One last time.

      Say it with me.

      "RAID is not a backup solution."

      Again,

      "RAID is not a backup solution."

      That being said, RAID helps to overcome failure of the drive. Do yearly or bi-yearly DVD backups, or back up to another offline harddrive... etc..

      • Re:Serious OS X user? (Score:4, Insightful)

        by Unholy_Kingfish ( 614606 ) on Saturday November 26, 2005 @03:58AM (#14117638) Homepage
        RAID isn't a COMPLETE solution, but it is a start.

        There should be two levels of file storage, a live copy, and a long term copy. EACH of these needs a backup. Once you need more room on the "live" server, you move it to long term storage. you want to keep as much "live" as possible to go back to.

        Live Copy - I would get a RAID system with as much storage as you can afford or need. If price is no option, then you get a few XServes since they are on OSX. This is where you keep your files from today on backwards till the RAID is filled. You keep a backup of this RAID by tape, HD, optical whatever.

        Long Term Copy - There are problems with EVERY type of long term storage. The most reliable would probably be hard drives. You use two(or more) NEW and different brand drives, copy the data to be archived to each, verify each. Store each drive in a static bag and some sort of case. Put one at your house, one in the bank or somewhere else. A bank's safe deposit box might be expensive, but it is climate controlled and "safe". Using tapes and optical disc are problematic because the mediums break down with age. The hard drives will last much, much longer. Yah, hard drives fail. But if you use different brands, you increase the chance of a mfg being better than another.

        I have HD's that are 15 years old that still work fine after thousands of hours of use. These archive drive will only be used long enough to format, test to make sure they aren't DOA, and to write data to. They should outlast the drive interface technology that they use.

        As time goes on, drive get bigger so more data can be live, and archiving becomes easier. You can always go back and re-archive the data to a "better" medium that holds more.

        Back in the day, people would back up their 20MB HD's to floppy, then they bought a 100MB drive, used tape to back it up, then CD, now DVD. Times change, so your backup strategy must change too. What you do TODAY, will be easier in a few years.

        • I've got punch cards that you can't even read the printing on that still hold the data perfectly uncorrupted; at least 25 years old. No mechanical device is ever going to beat good old punched cards or mylar/paper tape for longevity

      • Re:Serious OS X user? (Score:3, Interesting)

        by egarland ( 120202 )
        "RAID is not a backup solution."

        No.. but it is many many times less likely to fail in comparison to storing a single copy on DVDs which is what it is being compared to here.

        In general though, the need to do backups is greatly reduced with RAID and in some cases, a single copy on a RAID array is "good enough" which is to say it eliminates the need to create a separate backup copy. To claim otherwise is to not fully understand data backup.
        • To claim that RAID is "good enough" in any circumstances where data loss is unacceptable is to not fully understand data backup.

          What happens if two of your drives fail simultaneously? This is not as unlikely as it sounds, since those drives were almost certainly manufactured, bought, and installed more or less at the same time, meaning they'll meet their MTBF at the same time. Yes, the M in MTBF means "Mean", but they're still all going to be reaching the end of their expected life at the same time. And wh
      • I agree that RAID is not a full backup solution, but for a photographer it's the most important component of one (well, alongside an offsite backup or ten).

        The reason is that the data a photographer is storing is non-reproducable in a way that very little data is. If I spend a month programming and then loose all that code in a crash - a terrible annoyance, but because the idea of what to do is in my head I can re-create it with some work.

        But a photo gone is lost forever. Additionally the work that can go
    • He should get an Xserve RAID, of course.

      I'll second that! If he's already on an Apple environment the Xserve RAID is really the best choice. They're ridiculously easy to set up and they're surprisingly fast for still using ATA-133 technology.
            Of course I'd recommend a decent-sized tape drive (or possibly firewire hardrives) for off-site archiving as well...

  • Just answered (Score:3, Informative)

    by billster0808 ( 739783 ) on Saturday November 26, 2005 @01:11AM (#14117047) Homepage
    Try one of these [slashdot.org]
  • If I were tasked with this same job this is the solution I would use:

    http://coraid.com/ [coraid.com]

    Basically, you add your own disks, and have up to several terabytes of RAID storage. The best part is that teh RAID and all the complicated stuff is ghandled by the drive unit, to the OS it just looks like one huge drive.

    You can add a NAS (SAMBA/NFS) server (or roll your own), to make accessing the drive from Windows / Mac even easier.

    I don't have one of these myself, but have been drooling for a while...

    -Ms2k
    • This is definitely a good recommendation. ATA over Ethernet has comparable performance to a similar SCSI configuration. Additionally the price tag is not quite as hefty. Coraid's prices range from $2,000 for 2 Terabytes - $4,000 for 7.5 Terabytes. That does not include the price of a NAS server. Like the parent said you should be able to homebrew this. You could possibly set it up on the Mac G5 you currently use. Making the assumption that you can easily set up an NFS server in Mac OSX.

      If storing all

      • Re:ATA over Ethernet (Score:3, Insightful)

        by Proc6 ( 518858 )
        "Homebrew" and "critical data".

        Two phrases I don't like seeing anywhere near each other.

        Aim more for "redundant" and "widely tested" for starters.

        • 'Widely tested' is a checkoff point for something general purpose, that will be used in many different settings and applications. Reliability is a more important measure.

          'Redundant' implies there will be failure and that the only solution is to throw extra hardware (and money) after it.

          'Homebrew' is a term often used to describe a solution that is hand picked and configured. Not always a bad thing.

  • Not much to go on (Score:3, Informative)

    by egarland ( 120202 ) on Saturday November 26, 2005 @01:28AM (#14117113)
    2GB/session isn't really enough information to design a storage solution but I'll dump out some generic big, reliable and cheep storage suggestions.

    For large scale reliable storage I dislike both optical and tape. They both quickly become more work to manage than it's worth and have serious reliability issues. Hard drive based is the way to go and since hard drives do fail and that is a bad thing, it's best to use RAID. It's especially a good idea since RAID is getting easier, since hard drives are getting cheaper per unit and since SerialATA is making it easy to hook them up right.

    Heres a basic design that I'm actually working on for a home server for myself:
    http://secure.newegg.com/NewVersion/WishList/WishS hareShow.asp?ID=1764600 [newegg.com]
    It's a 3U rack mountable 2TB storage server. Put a Linux distro on it with some small RAID1 boot partitions and a software RAID5 storage partition, throw samba and some email-home config to notify of drive failures and you've got a decent place to store up to 1000 of those 2GB sessions. Zip up the old ones if needed for more space. If rack-mounting isn't desirable there are cheaper desktop cases that would probably be appropriate.

    If this is overkill a 4 drive RAID5 array or even a 2 drive RAID1 array is much much easier to accomplish. Standard case, motherboard, power supply and drives with a Linux distro and you're done. Hardware RAID is also an option but since software RAID's high CPU usage wouldn't be an issue here I'd go that route.

  • Pay for it -- it's a business expense (Score:5, Insightful)

    by plsuh ( 129598 ) <plsuh&goodeast,com> on Saturday November 26, 2005 @01:30AM (#14117121) Homepage
    This is a system for a professional photographer, storing the digital photos that are the lifeblood of the business. (Note: most professional photographers charge a nominal fee for a session, but then make the real money by selling prints. No negatives or no files = no $.) This is most emphatically NOT the place to try to do things on the cheap. It's an absolutely necessary and tax deductible business expense.

    I work for Apple, and while I'd prefer that this place purchases an Apple-based solution, I am not wedded to a particular OS or brand of hardware. However, you get what you pay for -- either through hiring a skilled professional building an open-source based storage system or by paying for a commercial solution (such as Apple's XServe RAID [apple.com] unit). Be sure to include the necessary system maintenance in the budget for such a complex setup, including off-site backups, on-call support, and making sure that it stays up and running during successive system updates and upgrades.

    Given that the photographer is already using an Apple G5, I suspect an XServe RAID solution will suit the situation quite well. One unit can provide 7TB of storage, which at 2GB/session works out to about 3500 sessions at current resolutions (also allowing plenty of headroom for growth as resolutions increase). Apple offers professional services [mailto], on-call support [apple.com], and training [apple.com] for server administrators. In addition, if you're looking for an Apple consultant with the necessary skills in your area, check the Apple Consultants Network. [apple.com]

    --Paul

  • I shoot a lot of photos and I ended up buying a couple external harddrives. They're certainly more stable in the long run than DVDs and it's easier to organize and view the photos than if they were on hundreds of DVDs. Almost all external drives have USB 2.0 or Firewire connections, so moving them onto the drive isn't too painful.

    If I did photography professionally I'd look at an external RAID solution. It's too expensive for a prosumer like me, but a pro should be willing to pay for something like that.

  • don't over complicate it (Score:4, Interesting)

    by tolldog ( 1571 ) on Saturday November 26, 2005 @02:48AM (#14117455) Homepage Journal
    buy an external 300 GB firewire drive every couple months, label the drive by the time period. If you are really worried, have 2 every couple of months, they are pretty cheap. Drives don't fail sitting on a shelf, at least not like dvds do. The last thing you wan't is a raid system thats active every day, it only increases the likelyhood of failure. raid is great for data you must access now, but a waste if you are just using it to back up data you only need once every so often.

    If it is really important, use tape backup, make redundant copies, and send one off to a data storage place. As others noted, a backup solution should be part of the cost of the job, and is not really that expensive when divided over the different projects.
    • Hard drives do fail while off. A single copy will yield data loss, it's just a matter of time.
    • Drives don't fail sitting on a shelf...

      You've obviously never pulled out a spare/backup drive only to see it not spinup. Hard drives can and do fail. It seems to me that a reasonably cheap solution is to extend the current dvd backups that they're doing to 2 or 3 copies of each disc. And burn them at a slow speed to ensure that the integrity is good. And possibly use parity files with them.
  • I've always been curious... any recommendations on how to browse that many pictures in a reasonable way?
    • Iview MediaPro [iview-multimedia.com] has been recommended to me. It's smart about offline files, supports all sorts of tagging and searching, and isn't scared of terabyte-scale archives.

      That being said, Gallery 2 [menalto.com] has most of the same capabilities. It's a web photo sharing package, but you don't need to give the whole internet access to it. Gallery 2 is quite a powerful database, and if you're smart about tagging things as you add them, the search functions are impressive.

    • Argh, I typed up a very lengthy response and slashdot ate it (something about a form key or something, I was too upset about loosing my post to care about the error message)...anyway, I use Picasa [google.com] which is now owned by Google and free. I'm not a professional photographer; all I photograph are places we go and things we do and the obligitory friends & family photos as we're out and about...but I do have close to 10,000 of them.

      Picasa allows one to enter keywords (IPTC keywords, actually, so I could wri

  • I' more or less in the same situation. I'm not a professional photographer or anything like that, but I do like to take lots of photos when I go somewhere. And since I do not trust CDs or DVDs for long-term storage, I store them on magneto-optical disks. It's more expensive than DVDs, but *MUCH* more reliable. I use 1.3GB disks, and they cost 1000 yen each (around $8.5).

    Since your friend has around 2GB of data on each session, he could get a 2.3GB MO drive and use 2.3GB disks. These cost around 2000 yen ($1
  • Not to mention it would consume a lot more power than is needed. Has the photographer been using Taiyo Yuden media? Or just whatever Maxell or Verbatim was on sale at Office Depot the week he was running low? Maybe he just needs better-quality media, and make sure the friend is burning no faster than at 4X.

  • What? (Score:3, Funny)

    by Kickasso ( 210195 ) on Saturday November 26, 2005 @03:26AM (#14117561)
    Nobody proposed the "a gmail account per session" yet?
  • to add a few cents here..

    i would definitely consider backing up to hard disk. avoid dvds like plague! i backup to two hard disks for redundancy but i do not do it in a raid set. raid is hardware AND software reliant in all forms and a hardware failure can leave you with a lot of lost data(raid controller failure, model not made anymore!) and software raid can fail via simple glitches and leave the data scrambled and data forensics cant pull usable data off the disk! so i suggest 2 firewire disks and tak

  • Well... (Score:3, Informative)

    by megaversal ( 229407 ) on Saturday November 26, 2005 @06:49AM (#14118046)
    Most everyone is recommending hard drives and I'm definitely part of that crowd. Most everyone says "RAID" as well. I'll tell you what I do at work, not for photographs necessarily, but for all our data. I have two servers for user data, one on each side of campus. One is the "active" server with RAID drives, the other is a backup. Each night (I have the luxury of a quiet network at night), I run a network backup to toss stuff over to the second server with RAIDed drives. This prevents the accidental "rm -rf" users that just 1 server with RAID wouldn't prevent against (of course other types of attacks WOULD kill two server solution, which is why important data goes to an external firewire drive from the backup storage server).

    Granted, not everyone has the cash to blow on all this, but my stepfather, who is also a professional photographer, has finally taken that all important step toward moving to digital. He's been backing up to CD and he usually gets away with a session on one or two CDs, not counting any editing he does in Photoshop (he still prefers to touch up photos by hand). Anyway, he has been watching his bookshelf fill up with CDs, much like all his file cabinets that store all his old hard copy negatives and select prints. Any long time photographer probably deals with the same stuff, which was a problem before digital ever came around.

    What I've been working with him on is what will most likely be a big storage server... even at 2GB a session, you could shoot every day of the year and only use 700GB, which will cost you about $300-400 in a non-RAID solution nowadays (based on me just purchasing 4x 300GB drives at $110 each and my friend buying a 400GB for $200). A small server with a few drives will be all the online backup one should need, plus to be extra safe, either that backup server, or just a few external drives.

    If you backup to the external drive once a week or so, this should save anyone from the accidental rm -rf (my stepfather once deleted all the pictures on his laptop by accidentally dragging the wrong folder to the Recycle Bin -- naturally all his photos were too big for the trash and were instantly deleted, luckily he had all his CDs to restore from). Plus, as long as you're backing up regularly, it should be obviously that the hard drive is working or not working. If you start hearing clicking, or feel something funny -- get it replaced.

    I guess my summary of all this is to have two backups. If one is your "online," primary storage, it should be obvious if it's failing or not failing, and assuming you're backing up to your second backup regularly, there shouldn't be any danger of you not realizing it's failing, because you are using it all the time. with DVDs and CDs and other media of that type, it's because you set it on a shelf and forget about it for years that is where the danger is caused.

    Sorry this was long.
  • Not sure which kind of photoghraphy your friend is doing, but don't sweat the details. That shouldn't be your job. Picture archiving is something magazines/agencies do professionally, and can throw serious amounts of money at a problem. Friend of mine is a picture editor, he's just organised a 10TB server, complete with backup, for the 17 photographers that work for his magazine. Agencies are even better in that respect, as they work for you and not the other way round, and are obliged to give you access t
  • I propose 35mm film.

    No, really. It has a really high information density.

    We could take the image and write it to film in digital form using optical drive technology...
  • use DVD-RAMs. They are much better engineered, without needless backwards compatibility and designed for long term storage. They're also a bit more expensive. If it's important enough, always have at least two copies of your data. (I guess, some king of ECC over many DVDs would be possible, but I doubt that anyone has ever implemented something like that.)

    Other than that, as others have already pointed out, you can always buy harddiscs. It's not what they're designed for, but still the easiest solution
    • You should look into SmartPAR, and the PAR file concept in general. You're generating FEC blocks for archives before dispatching them to [transmission | storage], so if one block is unreadable, you can recover it.

      Slyck [slyck.com] seems to have a good explanation of how this works. They're geared towards filesharing, but the concepts are useful for backups too.

      You'll need a lot of temp space. If you're filling 4.7-gig DVDs and doing 1 parity disk for each 5 data disks, you're talking about archiving a ~20-gig batch of
      • Interesting, I didn't know of this particular implementation. Slyck's explanation only covers simple parity, which is only capable of recovering a single missing block. An explanation of more sophisticated codes can be found on http://www.eccpage.com/ [eccpage.com]. SmartPAR is probably using a Reed-Solomon-Code.

        In principle, there's no need for temporary storage for the whole batch of data. Storage for all parity files would suffice and still allow single pass coding. Even better, given online storage for the redund
  • This guy is a photographer, not an administrator.

    Buy and label pairs of drives. HDs are cheap these days. One is the original store and the other is the backup.

    Name the first drive as driveA and driveAbkup, second set driveB and driveBbkup.

    Keep a notbook for eack to note the customers/jobs in each drive. This way he does not have to attach the drive to know which customers jobs are on it.

    When ever something on a drive is modified, use your favorite backup program to copy from driveA to driveAbkup.

    I currentl

  • Don't be innovative: Go with tape. (Score:3, Interesting)

    by hey! ( 33014 ) on Saturday November 26, 2005 @12:01PM (#14118855) Homepage Journal
    If the imperative is protecting his data, then he should do what professionals do with any other kind of critical data: put it on tape. Sure, you probably could put together an optical solution, but the tape technology is compact, proven, convenient, stable and scalable. I'd avoid anything proprietary aimed at the consumer level (if anybody still IS aiming tape technology at consumers), and look at technologies such as DLT which are popular for critical applications among professionals. Support for technologies adopted this way is measured in decades.

    Given that this is his life work, he really should invest a few thousand dollars and put together a strategy that will protect him from media and system failure, localized disasters such as fires, and possibly even regional disasters. With a little thought, while it is not going to be cheap, it will be a bargain.

    Supposing he's willing to put four or five thousand dollars into this. He can get a SDLT tape drive with a 160GB native capacity (don't count on compression for photos), and 16Mb/s native transfer rate. That day's photo session takes two minutes to back up. am deacj tape stores possibly up to half a year of work. He'll have enough money to buy a good number of tapes, so with a a little thought he'll have a good system for archiving his old stuff, one that is not vulnerable to single tape failures and has an offsite (important!!!) component too. And he may have enough money left over to buy a fire resistant media safe that could buy his data at least a couple of hours of time. Depending on the economic value of his work, he could also send backups to an offiste media storage facility that provides a very high degree of security against regional disasters as well.

    I'll tell you a story I tell all my clients when the cost and inconvenience of a well designed backup program comes up.

    Years ago I had a client who drove up with what looked like a huge piece of burnt toast in the back of his nice Mercedes sedan. He was was a CPA, and this was three weeks before tax day; the burnt toast was a minicomputer that had all his client's tax work on it. He'd been doing backups daily to tape, but contrary to our advice he had stopped bothering to take them off site. Under the circumstances, if he'd had an offsite backup, we'd have lent him everything he needed, even the office space if necessary. He'd have been back on track with maybe two days down time on the outside. When tax season was over he could have moved to a new office, bought new equipment from the insurance settlement, and his biggest worry would be decorating. But all this depended on the offsite backup he didn't have.

    There's a small chance that some of his data mightbe retrieved nowadays, by firms specializing in this sort of thing. But they didn't exist in the early 80s, an in any case I wouldn't want to bet on it. The computer had obviously taken major heat; the interior wiring and connectors weren't just smoke damaged, they were brittle from cooking. We did the best we could, removing the drives, stripping and swapping the electronics on them, cleaning all the connectors on the drive with tetracholoride and so forth. After a few hours of work it was clearly futile, but we spent another day on it trying pointless and hopeless things, just to make him feel like we'd done everything possible. None of this would have been necessary, but for want of a simple step he was fully equipped to take, but seemed like a bit too much bother at the time.

    The lesson is that while people comprehend small disasters like misplacing a file, large disasters are sometimes so horrible to contemplate that they discount them altogether. If your client is lucky, he'll be irritated with being saddled with having to swap tapes every morning and perhaps rotate them offsite every few days. Maybe labelling the tapes will be a chore. If he's unlucky, you'll be a hero.

  • As a general backup solution, I still have files from 1984 that are readable. These go back to my junior high school files from an Apple ][e. Obviously this method has been refined over time, but it works. It takes some work, but I'm paranoid about losing old data.

    1. Use high-quality brand DVD-R blanks. Currently about 35 cents/4.4 GiB.
    2. Verify all writes to make sure it was at least written correctly. Long term stability is unknown at best.
    3. Make multiple copies and store them in different locations. I store
    • Scarhead nailed it. I do most of these things, even creating par2 files.

      The key steps are 2) verify that your writes are correct and 3) make multiple copies and store them in multiple places. These steps are important no matter what media you use, though verification is less important on hard disks (since they're far more reliable than optical media.)

      Personally, I don't really bother with high qualify DVD-R media -- I go for the cheap ones. Though to be fair, I do throw away a fair number of them

    • # Use high-quality brand DVD-R blanks. Currently about 35 cents/4.4 GiB.

      Sorry, but I'm afraid the two are mutually exclusive. "High quality" media doesn't come at a bulk price of $0.35/unit. Generic consumer-grade media does. This is a person who's backing up thousands of hours of unreproducible work, not his MP3 collection.

  • Real Easy, Secure, and safe.. The Buffalo Terrastion, I just got on for this very purpose: http://www.buffalotech.com/products/product-detail .php?productid=97&categoryid=19 [buffalotech.com]
  • This system'll get you out of trouble for about a full year, given ten photo sessions a day. You'll get free on-site support and you can store a full session in cache!

    http://store.sun.com/CMTemplate/CEServlet?process= SunStore&cmdStartWebConfig_CP&familyCode=SE6920&ba seSelected=3 [sun.com]

  • The solution... (Score:3, Informative)

    by joto ( 134244 ) on Saturday November 26, 2005 @06:07PM (#14120390)
    This [ibm.com] is the solution. If you doubt me, look here [enterprise...eforum.com] for an article about it.

    It works. Reliably. In my previous job, we pretty much depended on it. A single faulty tape could cost us from $50k and up. And we didn't do backups of data on it... The tape drives were used continuosly 24/7.

    If you can afford it, is an entirely different question. I think it's about $30k...

    • Ah yes. These are nice tape drives. I'm in the market for one!

      If anyone has a 3590 kicking around please send me an email with details. I like the 256 track units. They are fast and realiable.

      As for $50K per tape - that is cheap.

      In the 128 track models the Fujitsu drives were more relaible than the IBM drives.

      BTW - if anyone needs some 10 tape autoloaders for the Fujitsu's I have several available plus main boards in both differential and single ended SCSI as well as some model H.

      Tape RULES!!!

      I'll never
  • I'm noting similar responses to a recent article - A question of stability of optical media [slashdot.org] was answered by some backup techniques. To repost: I really like parchive2 [sourceforge.net], but I wonder if dvdisaster [dvdisaster.com] is faster & allows finer-grained recovery, though.

    Someone else already posted about having offline hard drives...
  • I had some problems with DVD storage storing scanned family photos I can't replace. I found out it was the el-cheapo DVDs I was getting by the spool from CompUSA. Now, I use Verbatim MediDisc DVD-Rs. Not as good as the Hard Drive solution, perhaps, but much cheaper. These are used to store medical records and imagery, and I've been told that they are manufactured to much higher quality standards than the average DVDs. Not too much more expensive, either, especially in spools of 100.

    On a side note, for those
  • Get a very large server case with plenty of cooling, get a raid 5 card and make a raid array. Add more as needed. If you are going to store to a CD/DVD medium in addition to that, make parity files with something ala par2 as well, and get one of those food saver bags to chuck the jewel case in and store it in a vacuum in a dark area.
  • Tape will last longer then a DVD-R if taken care of.

    Actually, if the data is important id do both. 2 DVDs and keep the copies onsite, then a tape that is moved offsite to be stored *properly*.

    Sure, its a slow process to get your data back, but if you really, really, dont want to lose it..
  • RAID server

    5 disk RAID-5 would be good, lots of storage capacity with only 20% overhead for redundancy. should be doable for under $1000 and store a great deal of images
  • Please tell me more about that.

    I am sick unto death of cheerful articles that assert that [optical storage medium o' the month] has been proven in accelerated-life testing to last for umpteen aeons, and then discovering that my three-year-old disks can't be read... ...and saying "use a reliable brand," but no two people have the same opinions of what's reliable... ("Mitsui Gold or nothing..." "Just use the cheapest you can find..." "The Staples house brand is OK..." "No, no, it has to be a brand name like
  • Aftre trips I have between 2-10GB of photos to work with - that's before postprocessing which can add quite a bit of space as well. I also have a G5.

    What I have done is bought an external SATA enclosure for two drives, and a four-port SATA card. I have two 400GB drives set up using the software RAID in Disk Manager, configured to be Raid 0 (mirroring).

    The idea here is that I want enough space to hold all the photos I have ever taken and the work I have done on them. I only use two of the four ports for t

Slashdot Top Deals

He has not acquired a fortune; the fortune has acquired him. -- Bion

Close