Creating Large, Safe, and Cheap Network Attached Servers? 30
davco9200 asks: "I am looking to create a large data server for all my digital media files. The usage is the 'pro-user' category, to use the media from multiple stations in my house and at work. I value space (150+ gb would be nice), accessible from multiple platforms (Win, Mac), but perhaps most of all, some security (e.g. RAID 0 doesn't cut it). Total write or read access isn't that high of a priority. I have looked at things like the Snap 4100 that offer 160 gb or 300 gb and good raid options but the price seems high ($2,700 and $4,500 respectively). Has anyone had any experience making their own low-end NAS? Has anyone looked at the Adaptec IDE RAID Controller? This seems like a reasonable way of getting data parity so if one drive goes down your entire collection isn't lost. I figured Slashdot readers would have some good solutions. Information on specific cases, drives, and other pertinent facts would be helpful."
Well... (Score:3, Informative)
So you can cut the cost of a good IDE raid card and just put an extra IDE controller card in your psuedo-NAS box so you can have 8 drives. Or you can put that off until later.
You should also consider getting a DAT drive and a bunch of DAT tapes, to back things up, just in case something massively bad happens to your system. RAID 5 is not perfect, and if your system catches on fire because of too much dust in the power supply, it'll be helpful.
Re:Well... (Score:1)
I wouldn't let DAT tapes anywhere near backup of stuff I actually care about for several reasons, but mostly, capacity and speed
Firstly size, how much storage are you considering trying to backup? The biggest DAT tape you can get is 20-40Gb compressed. Now your post was talking about 8 Drives, 40Gb each, raid5 would mean 280Gb of data yep? so that would mean 9 DAT tapes to back the whole lot up (with an average of 30Gb per tape) that doesn't sound good to me
Secondly speed, how long will it take to backup 280Gb onto DAT? DDS4 is usually about 300Mb per minute (max), so that is 955 minutes - 15 Hours!
Now for a serious backup solution, even though it's expensive, you proabably want to consider DLT. You can get 40-80Gb on a single DLT tape meaning you only need 5. There are even DLT like solutions that will provide 100-200Gb on a single tape. Also consider speed, a good DAT drive can give you 300Mb per minute, a 80Gb DLT drive can do 720Mb per minute meaning only 6 Hours (one night backup run). Also you must consider reliability, DLT tapes are much more robust and are considered a decent archive media, I know you said cheap but you need to make sure you don't loose your data - that would be expensive.
Re:Well... (Score:2)
Bzzzt... we're working under a different set of rules here. I'm in the process of doing this exact same thing, and I'm using DAT for backup because they are perfect for what I want - long term, cheap, backup once and never again storage media.
Think of this way - who *cares* if it's a small box of tapes? Who *cares* if it takes 48 hours to back it up? I'm converting my entire CD library to MP3, and my entire videotape/DVD library to DivX ;-). As I get about 30-35 gigs of encoded data done (which takes quite awhile to encode), I back it up onto a single DAT tape. That's 50 movies, or 3500 songs... which will free up two shelves of videotape (or really one shelf, since I have to double shelve tapes), or a chunk of one of my 500 disk CD-changers, and turning two shelves of VHS into a very very small, durable tape is very very nice.
As you can probably guess by that last sentence, I have lots of media... tons of CDs and video. I have this fantasy that I get raided for "piracy" because I use MP3 and DivX ;-), it hits the news bit time, and I walk into court with dozens of boxes full of original videotapes, DVDs and CDs. Hehehehe.
My prototype system is a 450Mhz Pentium II with a Promise IDE RAID, with one Maxtor 80 gig hooked up, 64M RAM (I had two 20 gig hooked up to test the RAID earlier). I'm looking at upping the processor because encoding just takes too long, the memory seems fine for single playback or encoding, and the controller seems fine for 4 drives, but I want 8, probably Maxtor 80 gigs, as they are cheap.
My biggest problem is the initial video grab or analog sources. I haven't found a really good capture card, and I want one that runs under Linux, preferably with open source command line utilities so I can automate the process. As it is, I can convert DVDs just fine, and I have an All-in-Wonder Pro sitting on my desk that I want to test (since I have it, I might as well try it). I've seen some absolutly incredibly encoded porn out there - very clearly grabbed by an amatuer, and from a video (you can see the artifacts), but the clarity from an NTSC signal is incredible. I'm really curious as to what equipment is available for the prosumer for video capture, preferably Linux friendly.
Alas, as I just laid myself off from my company (at least for a few months until we start making better revenue), it dosen't look like I'll finish my media server for another several months. Maybe then I'll get 120 gig drives.
--
Evan
Re:Well... (Score:2)
Very good... and if I were a company with money to spend, needed to save time, or if I was a millionare masturbating with my system stats, I would get one. As it is, I've had the DAT drive for years, the media is dirt cheap, and HP's units are very reliable.
What part of "define your operating needs" didn't you understand in all this? We're looking at a homebuilt system out of cheap and available tech. I'm willing to trade time for money (as it is, a 450Mhz PII is perfectly fine, even if it's a "drop DVD in, and the next day, you've got a DivX ;-) file" situation or "drop a CD in, and do other stuff while it takes its merry time to encode"). While a more sexy bulk storage solution might net me space and time savings, I don't *care* if I've got a tape briefcase full of DATs versus one unit. Hell, for that matter - if that one media fails, I've potentially lost it all. If a DAT fails, I lost just part of the collection, and reencode just those. Speed of backup does not matter. Once backed up once, I don't ever do it again (although when it's all done, I'll probably make another backup of home videos and a few others and toss 'em to my relatives for safekeeping - cheapman's "off-site backup storage").
If I wanted to do it sexy, I wouldn't have tossed two 32Meg dimms in there, used a dirt cheap Promise RAID controller, or encoded to DivX ;-). As it is, the VHS tapes will go nicely to DivX, and I'll probably keep the DVDs on hand because DVD to DivX ;-) is a noticable loss in Audio and Video quality on my HDTV projector with a Kenwood full fiber 7.1 sound system (even though I run video though a analog video line to smooth DVD artifacts - we call the direct digital to HTDV on a seven foot tall screen the "Artifact Detector mode"). Still, for light fare like Urusei Yatsura, where the original wasn't high quality, I'll just watch the DivX ;-). For things like Apocolypse Now, Clockwork Orange, or The Wall, I want the best damn audio and video, and I'll go to the DVD for that (or the laserdisc). I'm willing to lose a bit of audio quality on CDs in exchange for flexibility, but I still encode at 256k. This is a hobby project born out of spare parts and fulled by the availability of dirt cheap drives and RAID controllers... and a desire to reclaim two damn rooms worth of videos and CDs, and something like 15u of space on my primary A/V rack (CD changers).
--
Evan
Re:Well... (Score:2)
You'll learn when you get married to or live with a female geek, a combined income allows you to buy the nicer things in life... until you have kids (which I've avoided to date).
The only really expensive item in my entire setup is the projector, which was $5800 refurbished. And I got that primarily because I host Rocky Horror rehearsals and the South Florida Anime and Manga club meetings (SFAM), plus, while I don't watch TV, I really like movies.
--
Evan
Re:Well... (Score:1)
Quite frankly I don't care about smelling pistakes, you know what I meant, so did I, so did probably just about everyone else that can read english. Also this is only slashdot, if I was publishing a paper or writing documentation for software that was sold then it would be another matter
As for being sorry, yeah right, why don't you worry about things that actually matter - like solving peoples problems, not wineing because the people who provide the solutions can't spell. I suppose you'd complain about being saved by mountain rescue if someone had misspelt mountain on the side of the helicopter
why hardware RAID (Score:1)
Solution (Score:2, Informative)
Used parts make it cheap... (Score:2, Insightful)
Obviously SCSI is more expensive than IDE, but you get a little bit more. Just food for thought.
Don't be *too* cheap! (Score:2)
Keep the chassis, but replace the power supply. Besides, since this is a server this sounds like a good place to use one of those units with a built-in UPS, even if box is hooked up to an external UPS. (I've had them fail because I unwittingly overloaded them, because of poor designs that allowed me to accidently turn off the UPS protection but not the power, etc.)
Windows 2000 Software RAID (Score:2, Interesting)
I don't know if an MS product is what you'd want to use, but it's out there.
J.W. Koebel
Re:Windows 2000 Software RAID (Score:1)
Re:Windows 2000 Software RAID (Score:2)
3ware IDE RAID is the way to go... (Score:1)
3Ware Escalade Cards (Score:1)
Backup to Hard Disk (Score:1)
Last I checked, Fry's was advertising big disks at under $2 per Gigabyte. That's cheap backup.
use those 3ware controllers (Score:2)
I think the overall costs were around 2000 Euro
(controller, drive bays, BIG case, etc
why NAS (Score:1)
KISS
For home use, right? Just buy (a bunch of) gigundo SCSI drives, and cram them into an Intel system running your choise of free/open source OS. I'm fond of the *BSD family, but YMMV.
Run samba and you'll see the files as if they were on drives native to Windows/Mac.
Or am I missing something?
Re:why NAS (Score:2)
Homebrew Firewire NAS (Score:2, Informative)
Buy a bunch of cheap, identical IDE HDs, and put them in IEEE-1394 cases (~$150/ea.). Compile yourself some bleeding-edge Linux-1394 [sourceforge.net] support, plug in your HDs, run XFS [sgi.com] as the filesystem, and use software RAID [linuxdoc.org]. Because you said this is just for storage and media access, you probably don't need the currently limited FireWire hot-plug support [sourceforge.net] and possibly still currently limited RAID hot-swap support [linuxdoc.org].
For more on software RAID, IBM has a nice two-part article (1 [ibm.com], 2 [ibm.com]) on it.
Homebrew Snapserver 4100 (Score:2, Informative)
This is my receipe for an "homebrew" Snap4100
1) Get:
- 1U 4bays rack mountable chassis from Sliger Designs [sliger.com]
- 3WARE 6410 Escalade [3ware.com] IDE controller (Choice of 0/1/0+1/5 Raid) on a 90 PCI riser card
- 4 x 75/100GB ATA100 drives (maybe DiamondMax [maxtor.com])
- MicroATX mainboard [tyan.com] with NIC and Video integrated on board (invest in RAM not in processing power - 750/850MHZ should be more than sufficient)
- Minimum Linux [linuxlinks.com]/*BSD [freebsd.org] OS booting from a read-only 16 to 64MB flash IDE device [tapr.org], loading kernel and a customised Ramdisk [linuxhq.com] root filesystem, mounting Raid devices in R/W mode, starting SAMBA [samba.org] (and/or Netatalk [anders.com]).
A good starting point is Linux Bootdisk HOWTO [ibiblio.org]
2) Choose 0+1 Raid and you get quick and completely redundant 150/200GB storage that can survive the full failure of one disk.
3) Want remote grafical managment from a standard web browser? Go for Webmin [webmin.com] or SWAT [samba.org].
Pretty easy (Score:2)
Yeah, probably about half the people on Slashdot. ;-)
There's really nothing to it. Get a x86 PeeCee, 4 cheapo 80GB ATA drives, and Linux or xBSD. Put a bootable+usablesystem partition on each drive (since you never know which drive is going to fail first) and use the rest of each drive as a slice for a RAID5. You'll have a about 220-240 Gig of storage that can survive media failure. You can do it for somewhere in the $1500-$2000 range.
For larger non-media failures, you're still screwed, though. Backup technology just hasn't kept up. :( Unless you have Big Money to spend on this (and maybe even then), you will end up using many tapes for a single backup. That fact will tend to influence you toward not backing up at all, unless you are unusually disciplined.
For the exporting to the other machines, the stuff you need should come with just about any Linux distro. I use NFS (for Unix and Amiga) and Appletalk (for Mac). I suppose Samba would work for wintel boxes.
Raid (Score:1)
I believe this would be an optimal solution get 2 cards
RC = Raid Controller
4 drives
RC1-----RC2
D1------M1
+--------+
M2------D2
8 drives
RC1-----RC2
D1------M1
D2------M2
+--------+
M3------D3
M4------D4
You need to have the mirrors on different cards
you will get better read/write performance
data for a drive on 1 controller and the mirror on the other, and keep the mirrors and data split between the controllers Ide or scsi, I'm not sure what functions are needed on the card or even if the 3ware cards will support it. I'm not sure how to configure this on the SW side either. But for my new DB server I will configure it this way.
You can lose 50% of the drives(if they are the right 1's it not the data and the mirror)
with raid 5 you can loose 1 drive, then the spare(if installed) must recreate the data before you can lose another drive.
-- Tim