Experiences w/ Software RAID 5 Under Linux? 541
MagnusDredd asks: "I am trying to build a large home drive array on the cheap. I have 8 Maxtor 250G Hard Drives that I got at Fry's Electronics for $120 apiece. I have an old 500Mhz machine that I can re-purpose to sit in the corner and serve files. I plan on running Slackware on the machine, there will be no X11, or much other than SMB, NFS, etc. I have worked with hardware arrays, but have no experience with software RAIDs. Since I am about to trust a bunch of files to this array (not only mine but I'm storing files for friends as well), I am concerned with reliability. How stable is the current RAID 5 support in Linux? How hard is it to rebuild an array? How well does the hot spare work? Will it rebuild using the spare automatically if it detects a drive has failed?"
Re:Don't go with 3ware (Score:2, Interesting)
We've run several 7810s, 7850s in the past, totalling quite a few terabytes. All in all it's not too awfully bad, but the cards do seem to have trouble with dropping drives that don't seem to have any real problems (they recertify with the manufacturer's utility often with no errors).
If you go 3ware though, get the hot swap drive cages from 3ware. They are expensive, but it makes it much nicer.
Re:Stick with hardware RAID, mod this up! (Score:2, Interesting)
Moderators, mod this up!
Re:Advice: Get lots of RAM (Score:5, Interesting)
Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?
Furthermore, block sizes default to 4k, though you could go to 8k or 32k block size. At any rate, you don't need a gig of RAM to handle this.
Finally, XOR is not that expensive of an operation, and a 500Mhz CPU is going to be able to handle that faster that any but the most expensive controller cards.
So unless you are actually a RAID kernel developer, I don't buy your story.
Re:Here is a better question (Score:4, Interesting)
Other than that, 3ware has been decent for us. We are about to put into service a new 9500 series 12 port SATA card.
I wish I could say our ACNC SATA to SCSI RAIDs have been as reliable. We have three ACNC units, two of them went weird after we did a firmware upgrade that tech support told us to do, lost the array.
We call tech support and they say "oh we didn't remember to tell you when you upgrade from the version you are on, you will lose your arrays".
Re:Stick with hardware RAID (Score:3, Interesting)
Back when I was using a PII-450 as a file server, I tried out software RAID on 3 x 80 Gb IDE disks. It mostly worked fine - except when it didn't. Generally problems happened when the box was under heavy load - one of the disks would be marked bad, and a painful rebuild would ensue. Once two disks were marked bad - I follwed the terrifying instructions in the "RAID How-To", and got all my data back. That was the last straw for me...I decided that I didn't have time to watch rebuilds all night. Note that this may have been caused by my crummy Promise TX-100 cards, I never bothered to investigate.
I got an Adaptec 2400 IDE controller, and it hasn't blinked for two years. One drive failure, and the swap in worked fine.
If the data is important to you - go hardware. If you want to lean something, and have the time to play, then sofware is OK. Just run frequent backups! If the data is really important to you, buy two identical controllers, and keep one in the box for when the other craps out. Having a perfect raidset, with no controller to read them, would be annoying.
Re:Works great (Score:5, Interesting)
that tool checks the SMART info on the disk about posible failures..
I do a lot of software raids and with smartctl, no drive crash has ever surprised me. i always had the time to get a spare disc and replace it on the array before something unfunny happened.
do a smartctl -t short
read the online page of it:
http://smartmontools.sourceforge.net/
A example of a failing disc:
http://smartmontools.sourceforge.net/examp
a example of the same type of disc but with no errors:
http://smartmontools.sourceforge.net/exa
Software raid works perfect on linux... and combined with LVM the things gets even better
RAID5 (Score:2, Interesting)
Re:Advice: Get lots of RAM (Score:2, Interesting)
Re:Advice: Get lots of RAM (Score:3, Interesting)
Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?
That would depend on the nature of the write. If you're writing the initial data it's unlikely that you'll require reading. However when you go to update the date you may have to perform reads in order to calculate the parity required for the update.
Software RAID 5 is very reliable but does suffer a performance hit. Not because of the XOR computations like many here are suggesting. It's because each logical write needs to be translated into physical reads/writes...which consume time.
The beauty of software RAID, at least software RAID implementations such as Veritas, is that it allows you to spread the RAID across a number of controllers.
Listen to this guy...he knows more than the others who consider the XOR computation the slow link in software RAID 5. It's not.
Re:Stick with hardware RAID (Score:2, Interesting)
The uptime isn't the reason for using RAID at home. Data integrity is.
With RAID, I don't lose all my data (or, if I take regular backups, all the data since the last backup) in the event that a drive fails, as long as I . A good RAID-5 setup will give me better read speeds than a single disk, at the cost of some write speed. Since reads are generally much more common than writes on a home system, this is an overall win.
However, these days disks are big enough that a RAID 0 configuration is reasonable, and that's what I have now. I get better write speeds and similar read speeds.
In any case, backups are no substitute for a good RAID setup. In fact, I would argue that the home situation is much more appropriate for RAID, because there simply is no good backup solution for home use -- hard disks are orders of magnitude larger than any reasonably-priced backup medium you can find. Only businesses can afford the kind of backup solutions that are capable of backing up the amount of data that's typical on a home system today without burning through a bunch of backup media.
Downtime isn't an issue for home use anyway. But loss of data is. That's why RAID solutions without hotswap capability are perfectly adequate for home use.
Re:Don't screw around - hardware is better. (Score:4, Interesting)
Hardware RAID5 is fine if your sole goal is reliability. If you need even an iota of performance, then go with software RAID5. The 3wares have especially abysmal RAID5 performance, specially older series like the 75xx and 85xx cards. 3ware's admitted it, and something targeted for fixing in the 95xx series (haven't gotten my hands on those yet, so I don't know).
As for software RAID reliability, I find that Linux's software RAID is much more forgiving than even the most resilient of hardware RAIDs. I've lost 4 drives out of a 12 drive system at the same time, and Linux has let me piece the RAID back together and I've lost nothing. Was the machine down? Yes. Did I lose data? No. Compare that with a 3ware hardware RAID system where I lost 2 drives. Even thought I probably could have salvaged 99% of the data off that array, the 3ware just would not let me work with that failed array.
Also, on any reasonably modern system, the software RAID will be faster. You just have a much faster processor to do the RAID processing for you. The added overhead of the RAID5 processing is nothing compared to a 1-2GHz processor.
Software raid is more flexible (Score:2, Interesting)
I make lots of smaller raid5
If you do go wtih hardware raid, make *sure* you can do a resize of the raid without losing your data. A lot of cheap raid controllers don't allow this - you have to wipe out all your data in order to add another disk to your raid, which is usually impractical. And you have to assume you're going to expand it.
Also, make sure to turn off write caching on your drives. It's much slower, but write caching is dangerous, especially in raid configurations.
Re:Please! (Score:4, Interesting)
Not compared to $0.
You see, the typical budget RAID 5 builder just wants to store his collection of MPEG4s, MP3s, and other downloads or perhaps uncompressed hobbyist video. It's not a database, it's not a 150+ employee corporate file server, it's just personal. Performance is not a concern.
And if performance is a concern (say he wants / on these disks) then the cheap way to go is software RAID 0, 1 or 1+0 (aka 10) *COMBINED* with a RAID5.
For instance, I just built myself a new system with four 300gb drives and partitioned each one like so:
50mb -
1gb - swap
20gb
5gb -
For the 50mb, I made a bootable RAID 1 of four drives (grub can boot this, dunno about lilo)
For the 1gb swap, I made a RAID 1 with two drives and a RAID 1 with the other 2. Thus I have a net of two 1gb swap partitions, with redundancy so my system will never crash due to drive-induced paging errors. This is essentially a RAID 0+1, though I let the kernel's swap system handle the RAID 0 aspect by giving them equal priorities.
For the 20gb
For the 5gb
With the four equal-sized partitions that were left, I made the RAID 5 for
Don't you see what a great cost-effective approach this is?!?
Maybe you work for some company with plenty of money lying around for $160 RAID controllers. But I'm in business for myself, and I don't see the sense in spending money where it isn't needed. Besides, the flexibility of software RAIDs (per-partition, not per-drive) would be well worth it to me even if something like the SX4 were cheaper.
Re:Stick with hardware RAID (Score:5, Interesting)
This happened to me. The card was sorta still working... could read, with lots of errors usually recoverable, but writing was flakey.
Luckily, even after about 3 years, 3ware (now AAMC) [3ware.com] was willing to send me a free replacement card. They answered the phone quickly (no long wait on hold), they guy I talked with knew the products well, and he had me email some log files. He looked at them for about a minute, asked some questions about the cables I was using, and then gave me an RMA number.
The new card came, and my heart sank when I saw it was a newer model. But I plugged the old drives in, and it automatically recognized their format and everything worked as it should.
This might not work on those cheapo cards like Promise that really are just multiple IDE controllers and a bios that does all the raid in software. Yeah, I know they're cheaper, but the 3ware cards really are very good and worth the money if you can afford them.
Mod UP please! (Re:Vinum with FreeBSD) (Score:4, Interesting)
Anybody here remember Walnut Creek's huge ftp archive at "cdrom.com" which back in it's heyday of the late 1990's used to be the biggest, most highest traffic ftp download site on the planet? They used a combination of Vinum software raid and Mylex hardware raid to handle the load. I remember reading a discussion article from them once that until you get a totally ridiculous volume of ftp sessions hammering away at ther arrays, that Vinum was actually a slight bit faster than the hardware array controller.
Maxtor 60GB drives for $18 after rebate (Score:3, Interesting)
Office Depot had an 18th anniversary sale, and was selling Maxtor 60GB drives for $18 after rebate. Bought three for my personal test machines, and used my friend's addresses for the rebates.
I often hear bad things about Maxtor drives, but after a whole 40 hours use, they haven't failed once.
Re:Another source of true hardware RAID (Score:2, Interesting)
The gentleman is correct. I've used Arco in 2 systems that ran for years flawlessly. Except for a drive failure. Which made the peizo alarm become annoying and LEDs change state.
And thanks so much for bring it up, because you reminded me I had one of those tucked away, forgotten, brand new in the box. I will be putting it into service soon.
tom
I hate sigs, and refuse to have one.
PCI bottleneck (Score:3, Interesting)
I don't know if anyone makes PCI-X ATA-133 controllers (non-RAID), so in the final analysis it might be best to get a 3ware card with a 64bit connector and plop it in a long slot. Of course, you need a pretty nice motherboard for that. I guess I haven't gone shopping recently, but they weren't that common the last I checked (and everyone is going to head for PCI-Express shortly anyway).
Of course, it all depends on what you'll use the machine for. If it's just file serving over a 100Mbit network, there's no need to worry that much about speed. It's only a big deal if you're concerned about doing things really fast. I believe good 3ware RAID cards can read data off a big array at 150-200 MB/s (maybe better). My local LUG put a ~1TB array together for an FTP mirror with 12 disks (using 120GB and 160GB drives, if I remember right) about 2 years ago, and testing produced read rates of about 120 MB/s on a regular PCI box (I think.. my memory is a bit flaky on that). Of course, I don't think anything was being done with the data (wasn't going out over the network interface, to my knowledge, just being read in by bonnie++ I suspect).
Re:Don't run software raid... (Score:3, Interesting)
First of all, linux software raid has excellent autodetection. You need to set the partition identifier to 0xFD so that the autodetector can identify it. As many have mentioned, software raid has a huge advantage over hardware raid for recovery - you can disconnect the drives from one computer, hook them to another and the autodetect code will figure it out. I know this works because I've done it.
Second, for 8 drives and 2 controllers a card, you'd want four ATA133 adapters. Each adapter has, as you said, 2 controllers. You don't want to sue the slave channel, because that will definately kill performance.
Third, Don't install the OS on the raided partition. Don't keep anything fragile or irreplacable on the OS partition. If you want to backup the configuration, backup the configuration. There's no need to raid your boot drive, and if your boot drive fails you can trivially reinstall.
A cheap batch script is not an effective backup solution. What if files are locked or a file is backed up midway through a transaction? I readily agree that RAID is not a backup solution, but to putting any faith in a "cheap batch script" is profoundly naive.
RAID5 has the advantage that you only lose one drives worth of space to parity information. With eight 250 gig drives on a P3 500, its readily obvious that his goal is to inexpensively store a large amount of data with an effective mitigation against a single drive failure. Software RAID5 is an excellent solution for him.
Lastly, I'd recommend one of the intel gigabit cards, because although the drives will only read 50 or 60 megabytes/s, the whole point is moot if your network connection maxes out at around 10 megabytes a second. The client adapters, like the 1000MT is more than enough, and not that expensive.
raid5 + debian (Score:3, Interesting)
When i started out, firefox was loading in 2 seconds and it now appears to be taking around 4 seconds to load. At least i think those mesurements are ok. If you want real speed, i'd think about using raid01 as it seems 4 discs in a raid0 array would be faster than 8 in a raid5? I'm not too sure about that, but raid5 is significantly slower than raid0 apparantly. Also, using those other 4 discs to mirror the raid0 array could be more usful then raid5s parity/crc redundancy.
Re:Stick with hardware RAID (Score:3, Interesting)
Re:Heat will be a problem (Score:3, Interesting)
After I set it up for the first time, I had a drive die on me really quickly and noticed when I replaced it that it was murderously hot. As in "burning my fingers" hot. So I went and bought these little hd cooling fans that fit in front of a 5 1/4" drive bay (and come with 3.5" drive mounting adapters) and have 3 little fans on them. They cost about $7 each. I put 4 of them in my machine and they kept the drives at room temperature. Ahhh.
But the noise was a problem as all those fans together sounded like a wind tunnel. Especially 2 years later when all the little fans started dying and making extremely loud noises. Think annoying fan noise multiplied by 12. Ugh. Then I found this neat product:
Cooler Master 4 in 3 device module [coolermaster.com].
Instead of 12 little fans I now have one big super-quiet fan, and my drives still say nice and cool. It was definitely worth the $30 I paid for it.
Don't forget about heat.
-David
Re:Advice: Get lots of RAM (Score:3, Interesting)
Linux software RAID5 has a considerably better chance to work nowdays. There are very few controllers out there that have unresolved bugs. Off the top of my head here are a few:
As far as controller duty roster is concerned we should also mention Via. From being the worst controller for Linux once upon a time in 1997-9 it has become the best. I have been getting better IO performance on ITX with C3 then on Xeons with server controllers for some time now (starting from around 2.4.23).