Bulk Data Storage For The Common Man? 483
Vigyaan writes "Lately, I have been looking into different bulk data storage options available to a common man. My work
depends on generating, storing and analyzing a
large amount of data -- averaging about 1 TB per
month. I would like to have a storage system which is automated, fast, reliable
and most importantly does not cost the price of an
eye. Right now, I have a 4 node Linux cluster with
10 large hard disks (total capacity 1.6 TB); data storage roughly costs
about $0.60/GB (excluding the cost of PC
hardware). But long term storage is painful -- DVDs
cost about $0.10-$0.15/GB but takes too much human time
and leaving data on hard disks makes me nervous
because of possible failures. RAID is a possibility, but it increases the cost significantly. I was wondering, if
Slashdot readers have any recommendations for a
cheap automated way to store and retrieve data."
Waiting for ... (Score:3, Interesting)
Those will be sweet =)
Wirewire drives? (Score:5, Interesting)
Again, I'm not sure if that's as cheap as you'd want, but that's a solution I came up with for a similar problem. My company's going to be 3D rendering some stuff that could end up eating 50 megabytes a frame. (Extra data is stored for future refinement... I can go into detail if I've piqued anybody's curiosity.) We can't afford to lose this data, so the Firewire drive approach is what we're considering right now.
Well... (Score:4, Interesting)
Tape? (Score:2, Interesting)
Re:1TB a month?!? (Score:1, Interesting)
eh (Score:2, Interesting)
If you don't have the patience for DVD backups (neither do I), then you're pretty much stuck with RAID. So buck up, spend the extra cash, and setup a storage box or two on the network with one or two terabytes in each. I have a branch of my network setup on gigabit, one box has 250 GB of storage on RAID 1 across two 250 GB (this one's for video projects), the other has 160 GB in RAID 0 (my learning system). Works fine and easy as hell to setup. If I need to add storage I can either add some drives or just add another box. I've thought about using GFS, but I don't know enough about it to implement it, yet. Anyone here currently using GFS?
Use those HDDs! (Score:2, Interesting)
The stockmarket is backed up to three (or more?) seperate locations. Look into NVRAM (e.g. flash media) or a cluster with all those hard drives linked together, with a constant backup. With the builtin IDE controller on most motherboards, you can hook up to 4 Hard Disk Drives. If you add SATA, RAID, SCSI, and IDE, you can have lots of hardrives on one machine!
You could also rotate hard drives, so they arent constantly used (making the whole system last a LOT longer!) or replace the drives that are about to fail (which would be at least in 3 years!). Most Hard Drives could probably handle 5 to 10 years no prob (maybe even 20 if they are rotated!).
It all depends on what you have and what you want to do!
Network Appliance (Score:1, Interesting)
They aren't File servers, as they aren't designed for lots of clients. But they are perfect for storing a 'live' backup of data ! They can the Technology Nearstore, its designed to sit between your File servers & your Tape backups
CD Changer (Score:2, Interesting)
Now the problem is, you can only get 430gig's out of one changer using single layer dvd's... Double would bring you to 970gig's per changer.
Assuming you can get the unit for 100 bucks or so, and the dvd drive costing 100 (69 bucks at frys).. Then you have a 200 dollar backup unit that can store 430gigs of information onto dvd's
Consider Online Backup (Score:2, Interesting)
Re:Hard disks (Score:3, Interesting)
Re:Hard disks (Score:5, Interesting)
Re:Wirewire drives? (Score:1, Interesting)
Firewire hub with hardware RAID (Score:3, Interesting)
How expensive could something like this really be? $300-400 at most, I'd have to guess considering what most places are charging for SATA RAID cards.
Do It Yourself CD Changer (Score:2, Interesting)
Do It Yourself CD Changer [slashdot.org]
Re:Use those HDDs! (Score:4, Interesting)
That said, this method would still be more than twice as expensive as storing data on hard drives, would still require a million pages, but would take a little under 2 weeks to print.
It still doesn't seem like a feasible option.
The up-side is that, if stored properly, the data would likely be safe potentially for many hundreds of years.
Another perspective (Score:3, Interesting)
Just tossing out another point of view - similar but different than some of the others previously discussed. First off, examine the data you are keeping - do you really need that much? Nowadays it's common to be able to acquire data faster than it can be processed, and if you never stop gathering data, well, you never will catch up, only fall farther and farther behind.
If you DO need this data, and you are going to need it for awhile, (year or more) I'd recommend cheap HDs. They also have an advantage of being easily catalogged, and are untouchable when compared with access time of tapes. Don't go raid5 though, this is not "catastrophy-proof". (flood, fire, tornado, etc) For catastrophy protection, mirror your drives. When you have them loaded up with data, pull the FW cables and swap drives in the enclosures with fresh empty drives. Label them well, and then take each half of the mirror to DIFFERENT LOCATIONS. It's OK to keep one set on-site, but the other set must be somewhere else, preferably in another zip code. This will allow you near instant access to your data (since it's onsite), will protect your data from mechanical failure (through mirroring) and will protect you against catastrophy. (you WILL need to acquire new firewire boxes etc if your office gets leveled... don't forget this detail in general - the data is of no value if you lack the equipment (tape drives etc) to read it back in with) I know you can get compression and fit more on a tape etc by using archiving software, but it may be worth the extra cost to obey the KISS rule and just simply drag and drop the data to the formatted HDs. This will make data recovery MUCH SIMPLER, and if there are errors on the HD when you need to recover, this will insure you can actually recover most/all of the information. Archive streams and tapes are notorious for losing 100% of the data that follows a corruption point in the stream.
Once you know you no longer need a specific set, drop it back into the pool of usable drives. Buy them by the case, it's much cheaper this way. It also is advisable to buy the same make/model every time you have to get more drives, even if there are newer, larger, cheaper models out, because having all the same drives means one less complication to worry about in times of crisis.
Cheap USB to IDE adapters (Score:1, Interesting)
USB to IDE cable [sewelldirect.com]
Re:Finally a use for my 1GB Gmail invites... (Score:2, Interesting)
Re:Personally I prefer something in a blonde (Score:3, Interesting)
KFG meant to say "You can have fast, good, or cheap. Pick two."
It's an old software design maxim that applies suprisingly well to this subject.
...and to many things, particularly if you replace "fast" with "convenient". Just for kicks, think about it.
Food? Check. Clothing? Check. Beer? Check. Housing construction? Check.
Pretty much anything that involves the exchange of money for goods and services follows this maxim.
How about abusing physics law's? (Score:3, Interesting)
Re:Finally a use for my 1GB Gmail invites... (Score:3, Interesting)
You always need at least 3 generations of backup. The Current backup, the "father", and the "Grandfather." These are complete backups, not incremental. And you need them in case you run into a media error. In our case we keep the last week of tapes, a weekly backup from the last 30 days, a monthly backup from the last year, and a yearly backup starting at the dawn of time.
If the data isn't worth backing up properly, you might as well not bother backing it up at all.
Re:Cheap solution (Score:3, Interesting)
Indeed, I don't believe in any backup that doesn't have multiple copies that can be stored offsite. Fire really doesn't care what was on your hard drive, nor do thieves, or axe-wielding maniacs.
And anyone who has been in IT long enough can tell you one of the above stories first hand.
somewhat cheap solution, 3 x RAID5 (Score:1, Interesting)
Use 2 5,25" bays for a dvd burner and a tape unit. In the remaining 9 5,25" bays, assemble three Promise sata enclosures, these can hold 4 disks each, and are 3 bays high. Then plug three Promise S150 SX4 RAID 5 controllers with 128 MB PC133 ECC each. Now populate all the raid enclosures with 400GB Seagate disks, as soon as they're available, for a capacity of 1,1444 TB x 3 = 3,433 TB grand total.
Initial cost:
1 x computer = 1500$;
3 x S150 SX4 + 128mb = 900$;
3 x SuperSwap 4100 = 750$;
12 x 400GB HD, 200$ each? = 2400$:
total = 5500$, 1,60$ per GB.
Have handy some 4 unit sets of SuperSwap 1100 enclosures for additional 4 disk raid 5 sets.
If you use a mobo with integrated Giga Ethernet, you can expect a network throughput about 60 to 75 MB/sec.
This setup will provide you with online access to the current data set, plus two backups, and as many as you can pay backup sets in closet storage.
You can reduce the initial cost reducing online raid sets, 1000$ each.
Hope it helps,
Bug Eyed Hardware Nut.
Re:Hijack Cassini (Score:2, Interesting)
You laugh, but before Project Whirlwind (which created the physical modern computer as we know it) settled for 16-64 bit CRTs (DRAMs, I suppose? :-) and then invented 3D core memory, they seriously considered leasing a micro-wave line to do exactly that.
At the very beginning, people were really hard up for memory solutions; e.g. the first Univac model used mercury delay lines, a variant on this concept.
(The out of print Project Whirlwind: The History of a Pioneer Computer by Kent C. Redmond is recommended if you're interested in this area of history.)
Re:Give Up Now (Score:3, Interesting)
Sounds trite, but EBay to the rescue.
I started with a single Exabyte 8mm backup drive and picked up 2 more on EBay for around $150. (This was a few years ago even and the original drive had been given to me with 50-60 used/new tapes.) Now that I have a DVD-burner, those drives don't do me much good (too small of capacity, and way too slow compared to DVD).
Rule of thumb for corporate / business use is that you always buy 2 or 3 of any mission-critical backup hardware. That way, if one of the units breaks, you still have the other to rely on while the first is either fixed or replaced.
Having identical backup hardware and software at another location is also a great idea.