Experiences w/ Software RAID 5 Under Linux? 541
MagnusDredd asks: "I am trying to build a large home drive array on the cheap. I have 8 Maxtor 250G Hard Drives that I got at Fry's Electronics for $120 apiece. I have an old 500Mhz machine that I can re-purpose to sit in the corner and serve files. I plan on running Slackware on the machine, there will be no X11, or much other than SMB, NFS, etc. I have worked with hardware arrays, but have no experience with software RAIDs. Since I am about to trust a bunch of files to this array (not only mine but I'm storing files for friends as well), I am concerned with reliability. How stable is the current RAID 5 support in Linux? How hard is it to rebuild an array? How well does the hot spare work? Will it rebuild using the spare automatically if it detects a drive has failed?"
Advice: Get lots of RAM (Score:0, Informative)
The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive.
In theory, this could be done simultaneously--read from all drives at once. In practice, Software RAID and ATA isn't so good at that kind of thing. (Good hardware RAID is a different story.)
So the idea is that those six reads will take a reasonable amount of time, every time there is a write. If you have a lot of RAM, and/or don't write really large files, it won't be a problem because all the data can be cached in RAM and the reading/writing involving the disks can be done later, at the OS's leisure. However, if you don't have a lot of RAM, or copy really big files, you'll have performance issues.
You may not notice this for a little while, until your array starts filling up, because some implementations (not sure about the Linux software one) optimize it so that they assume unused sectors are filled with a known value, so they don't actually read from drives where the sectors haven't been written to yet (they keep a big table in memory). This is a GREAT optimization. But over time, it will get slower and slower.
So my advice to you is to install a lot of RAM in this system, whatever the motherboard allows. At least one gigabyte, but preferably two or more.
Works great (Score:5, Informative)
Stick with hardware RAID (Score:2, Informative)
Though from what I hear, software RAID on Linux works decently.
Where I used to work. (Score:4, Informative)
I'm doing this as well, my experiences: (Score:2, Informative)
However, when a drive did drop off line, unless things are on their own channel, it seems to knock off the entire IDE channel as well. It ended up taking the better part of a day to get everything back online again, without any data loss.
It even seems like any time there's an IDE hiccup, you can knock your array offline.
It's definitely cheaper than hardware RAID, and I haven't noticed any performance problems, but sometimes the stability of good old SCSI raid is something I miss.
Re:hmmm (Score:0, Informative)
mp3s
movies
scat porn
japanese tentacle rape dating simulators
video captures of his nubile, young 14-year old neigbor undressing, oh to view her budding sexuality she is but a flower of femminine innocence
MAME Roms
Works Great! (Score:2, Informative)
The best advice I can give is to make sure each drive has its own channel if you are on standard ATA, you didn't specify SATA or regular ATA. If you're using SATA then all the drives get their own channel by design. If you have more than one IDE device on a channel in a RAID, performance will suffer because IDE can't write to both devices on the channel simultaneously.
Re:Stick with hardware RAID (Score:2, Informative)
> Generally for situations where you really need to make sure
> the data stays safe, I'd just stick with hardware. If you can
> spend that much on some harddrives, I don't see why you can't
> spend the money on hardware.
Truer words were never spoken. I don't know the status of the more recent software RAID implmentation in Linux, but I do know that bugs in the old one send 2 arrays in 2 different mission critical servers of ours down in a hailstorm of fire and brimstone.
We had one drive get booted from the array for having corrupted data, so the load on the other drives shot up a bit. We think that the increased load made the software RAID driver start lagging in writes to the disks, causing more corruption on another drive, until we were down to a steaming pile of rubble.
Happened 2 seperate times on 2 different machines, as well. We're sticking to hardware from now on.
Vinum with FreeBSD (Score:3, Informative)
BTW, I switched from Linux to FreeBSD for the server years ago for the stability.
Don't screw around - hardware is better. (Score:5, Informative)
Re:Stick with hardware RAID (Score:3, Informative)
Stick with Linux RAID. It knows how to do it better.
*DO* go with 3ware (Score:3, Informative)
Re:Advice: Get lots of RAM (Score:5, Informative)
When one of the drives fails--and one of the drives will fail--this will allow you to swap in the replacement drive immediately, before another drive fails. (Remember, if two drives fail in a RAID-5 array, you lose data.) You can then return the defective drive, get a replacement from Maxtor, and when that one arrives FedEx in a few days, that one will be your new "spare."
You can either keep your spare drive unused, outside the computer, or keep this spare "hot"--in the computer, connected and ready to go, but unused by the array or anything else, and have the array fall over to it automatically when a drive fails.
Both ways offer advantages. If you keep the drive out of the computer, since you need to shut down to remove the bad drive, you can install the spare drive at that time. If you were to keep the drive "hot" in the meantime, your extra "new" drive has been spinning for months or years, and exposed needlessly to heat. Which increase its probability of failure, making it essentially as likely to fail as all your other drives that have been running the whole time.
However, keeping the spare "hot" means that the array can be rebuilt sooner, in some cases automatically before you know there is a problem. This can reduce the possibility of data loss. You will have to reboot twice--once to remove the defectie drive to return to Maxtor, and once when the replacement arrives to install it as the new hot spare.
Which of those two choices is a judgement call, but it's absolutely critical to have a spare drive on hand.
Or, have two backup RAID controllers. (Score:5, Informative)
This is a VERY big issue. We've found that Promise Technology RAID controllers have problems, and the company doesn't give tech. support when the problems are difficult, in our experience.
--
Government data compares Democrat and Republican economics. [futurepower.org]
Re:Please! (Score:5, Informative)
There is no such thing as a "cheap" hardware RAID 5 controller. Well there is, but they'll still set you back at least $120 and are crap.
There are RAID controllers from highpoint and promise, et al that are card-based, but they are still CPU bound (that is where the XOR really takes place). So they're really nothing more than a controller with a driver that does the calculations in the CPU. These cards are good for booting windows to a software RAID (since that is essentially what they are) but not good for anything else.
Most motherboards especially those with only 2 RAID ports (whether IDE or SATA) are software-based, as well. The nvidia nforce3 250 is one of the few notable exceptions.
But the bottom line here is: Linux Software RAID 5 is a logical approach if simple redundant mass storage is your main concern, and will save you at least $120. Also note that for RAID 0/1 it doesn't really matter if you go hardware or software since they aren't very processor intensive anyway. Pure software RAID 0/1 seems to be easier to set up in Linux (less mucking around with drivers) so it often makes sense to go with it for that reason alone.
Hot spares (Score:1, Informative)
nr-spare-disks 1
device
spare-disk 0
Performance Tips (Score:5, Informative)
If you've got a lot of data that is read/re-read or written/re-read by clients then RAM really helps, streaming stuff which doesn't get many repeat accesses (eg running a movie editing suite) it might not help at all
For performance its often worth sacrificing a bit of space and going RAID 1. Again depends if you need the space first or performance first.
Obviously don't put two drives of a raid set on the same IDE controller as master/slave or it'll suck. Also if you can find a mainboard with multiple PCI busses that helps.
Finally be aware that if you put more than a couple of add on IDE controllers on the same PCI bus it'll suck - thats one of the big problems with software raid 5 versus hardware which is less of a problem with raid 1 - you are doing a lot of repeated PCI bus copies and that hurts the speed of drives today.
I use raid1 everywhere, disks may be cheap but you have to treat them as unreliable nowdays.
Re:Advice: Get lots of RAM (Score:4, Informative)
IOW: Two reads, and two writes. Not six reads and two writes. But yes, large amounts of RAM is a good idea. Of course, if a drive goes south, everything goes out the window and your performance will be shot until you replace the dud drive and everything resyncs.
Re:Don't go with 3ware (Score:2, Informative)
I've done software RAID as well using Promise IDE controllers. Fortunately for us we never had a drive fail in the software RAID so I can't comment on how difficult it is to recover from a failure.
Interestingly enough, we ran some fairly intense iozone tests on both the hardware and software RAIDs with very little difference in performance (maybe that's why the parent poster doesn't like the 3ware stuff). But... we also ran these same tests with a fibre-channel SAN disk, again with very little performance difference.
Maybe it was a Bus limitation... I didn't have time to investigate it any further.
CONFIGURE IT RIGHT!! small parts... (Score:5, Informative)
First, ensure that all of the drives are IDE masters. Don't double up slaves and masters.
Secondly, DON'T create gigantic partitions on each oft he 250's and then RAID them together, you will get bitten, and bitten hard.
Here's the skinny...
1) Ensure that your motherboard/IDE controllers will return SMART status information. Make sure you install the smartmon tools, configure them to run weekly self tests, and ensure you have smartd running so that you get alerted to potentially failing drives ahead of time.
2) Partition your 250GB drives into 40 GB partitions. Then use RAID5 to pull together the partitions across the drives. If you want a giant volume, create a Linear RAID group of all of the RAID5 groups you created and create the filesystem on top of that.
Here's why, this is the juice.
To keep it simple, let's say there are 20 secotrs per drive. When a drive gets an uncorrectable error on a sector, it will be kicked out of the array. By partitioning the drive into 5 or 6 partitions, let's say hd(a,c,e,g,i,k,l)1 are in one of the RAID5 groups, which contain sectors 1-4 (out of the fake 20 we made up earlier)
If sector 2 goes bad on
By partitioning the disks you localize the failures a little, thus creating a more likely recovery scenario.
You wind up with a few RAID5 sets that are more resilient to multiple drive failures.
If you are using a hot spare, your rebuild time will also be less, at least for the RAID5 set that failed.
I hope this makes sense.
My advice to you is to bite the bullet and simply mirror the disks. That way, no matter how badly they fail you'll have some chance of getting some of the data off.
Re:Stick with hardware RAID (Score:5, Informative)
I disagree with this. Here's why: the most important thing is your data. Hardware RAID works fine until the controller dies. Once that happens, you must replace it with the same type of controller, or your data is basically gone, because each manufacturer uses its own proprietary way of storing the RAID metadata.
Software RAID doesn't have that problem. If a controller dies, you can buy a completely different one and it just won't matter: the data on your disk is at this point just blocks that are addressable with a new controller in the same way that they were before.
Another advantage is that software RAID allows you to use any kind of disk as a RAID element. If you can put a partition on it, you can use it (as long as the partition meets the size constraints). So you can build a RAID set out of, e.g., a standard IDE drive and a serial ATA drive. The kernel doesn't care -- it's just a block device as far as it's concerned. The end result is that you can spread the risk of failure not just across drives but across controllers as well.
That kind of flexibility simply doesn't exist in hardware RAID. In my opinion, it's worth a lot.
That said, hardware RAID does have its advantages -- good implementations offload some of the computing burden from the CPU, and really good ones will deal with hotswapping disks automatically. But keep in mind that dynamic configuration of the hardware RAID device (operations such as telling it what to do with the disk you just swapped into it) is something that has to be supported by the operating system driver itself and a set of utilities designed to work specifically with that driver. Otherwise you have to take the entire system down in order to do such reconfiguration (most hardware RAID cards have a BIOS utility for such things).
Oh, one other advantage in favor of software RAID: it allows you to take advantage of Moore's Law much more easily. Replace the motherboard/CPU in your system and suddenly your RAID can be faster. Whether it is or not depends on whether or not your previous rig was capable of saturating the disks. With hardware RAID, if the controller isn't capable of saturating the disks out of the box, then you'll never get the maximum performance possible out of the disks you connect to it, even if you have the fastest motherboard/CPU combination on the planet.
Re:Devil in the... (Score:3, Informative)
Which is absolutely horrible. This violates protocol - mail MTA's demand that data is written to disk before they acknowlege delivery. They get this from the confirmation from the kernel, but if the disk array lies about it, a power failure could lose data even though the kernel assumed it had bee synced properly.
Re:Stick with hardware RAID (Score:2, Informative)
for the trolls: a real raid controller for me is a HP/Compaq smartarray, IBM server raid, Intel RAID, some megaraid controller...
i have a Proliant DL380g2 at my home, and it has a smartarray 5i with 32MB RAM of it's own, and a risc CPU for the array computing...
doing any raid configuration on the machine... the system CPUs are not affected.
What is the definitive article? (Score:3, Informative)
Is this the definitive article about software RAID under Linux?
Software-RAID HOWTO [unthought.net]. In English and HTML: Software-RAID HOWTO [unthought.net].
--
Bush borrows [brillig.com] money to kill Iraqis [iraqbodycount.net]. 140 billion borrowed [costofwar.com]. With interest, you pay 200 billion. When Saudis attack, invade Iraq?
Re:Advice: Get lots of RAM (Score:5, Informative)
Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?
Unless you write across a whole row in the array, how are you going to compute the new parity without reading in something? This is the "small write problem", and it is why expensive RAID controllers have a non-volite writeback cache.
The current kernel does read in the whole row to recompute the parity for simplicity. Technically, though, you just need to read in the block you are modifying and the parity block, making writes take 4 operations under RAID 5, but unless something has recently changed, Linux doesn't do that. A gig of RAM, however, will allow a degree of volitile write-back cache, to help offset what will otherwise be poor write performance.
Software RAID on Linux (Score:5, Informative)
mdadm will allow a "spare pool" shared between multiple RAID devices and smartd will check the state of the disk controllers at regular intervals. You should put the system _and_ the disks on UPS to avoid losing data in the event of a power failure (the disks need to write their cache to the physical media before it evaporates). Set up something (mdadm or smartd) to email you in the event of a disk failure, or you may be running in degraded mode for quite a while before you discover it (unless you look at
All in all it seems to work fairly well if you spread the disks across multiple channels, if you have enough RAM for page (buffer) cache, and if you get reliable disks. I have a 4-disk SCSI storage box that I have in RAID 5 mode. It has been running for over two years. The server failed and I had to move it, that is when I discovered mdadm -- A LIFE (DATA) SAVER!
Re:Advice: Get lots of RAM (Score:2, Informative)
RAM is helpful on ly in the sense that it can cache data and make things appear faster if the data is already available in cache. It won't really help it read/write data any faster.
If a drive goes out, write performance is inchanged, as the XOR operation must be done no matter what.
Read performance depends - it's just an XOR operation, which is not very difficult. A 500Mhz CPU will still be mostly idle even in degraded mode. Some implementations *could* do the XOR in regular mode too, to check for data errors, in which case, no performance is lost in degraded mode anyway.
depends on what you need (Score:1, Informative)
Re:Advice: Get lots of RAM (Score:4, Informative)
Also, any half-decent RAID implementation will have that hotspare in the machine with its spindle off until it is needed. So it won't have been spinning for months/years at all. Not quite as good as having it in a box as far as wear and tear, but very close.
Experience (Score:2, Informative)
Re:Don't run software raid... (Score:4, Informative)
Um, that's bogus. If your OS goes (probably due to hardware?) then you can simply put the drive in a new computer (same basic master/slave setup) and away it goes. Linux knows how to detect its own RAID arrays!
OTOH, if you have a hardware RAID, good luck getting tech support, especially if they no longer carry that board, or have gone out of business altogether.
At least with Software RAID, your data is not stuck in a proprietary format.
Re:8 drives? Maybe I am missing something? (Score:3, Informative)
500 MHz? (Score:3, Informative)
If you're spending $960 for the disks at Fry's, why not spend another $80 to $250 at that same Fry's and get a current generation motherboard and CPU (they have package deals that are dirt cheap).
For $80, you can get a 5x faster processor, and a much newer chipset with ATA133 and Serial ATA.
For $250, you can get a board with multiple PCI busses, PCI-X and a chipset capable of handling much more throughput than a cheap PC motherboard.
The I/O bandwidth will be your bottleneck with an 8 drive RAID array. The standard 32bit / 33MHz PCI bus only does about 1Gbps. Serving a gigabit ethernet connection will use all your bandwidth by itself.. when you have 8 ATA drives fighting the NIC for bandwidth, you can see a clear problem.
If you're spending that much for the drives, don't hamstring it by skimping on the motherboard. And, in any case, once you have a Linux box installed, you inevitably start using it for many tasks (caching proxy, mail server, ftp server, dns server, www server, etc). So, a beefier system will stand up better.
Promise is SHRAID, not RAID (Score:4, Informative)
Good, relatively inexpensive IDE and SATA RAID can be had with 3Ware Controllers [3ware.com]. 2-drive models start around $140, and they support up to 12 drives on their more expensive controllers. The drives appear as a single physical device to the O/S, whether it's Windoze, Linux, BSD, DOS 3.1, etc.
Re:Advice: Get lots of RAM (Score:2, Informative)
There's a reason this is posted anonymously. It's absurdly incorrect. Disregard and move on.
Worked for me in production (Score:4, Informative)
Another source of true hardware RAID (Score:5, Informative)
Software RAID Experinces (Score:5, Informative)
I manage a lot of servers remotely. I started out using the hardware RAID support on my server's mobos. But there were issues with that.
First, it was hard getting Linux driver support (I think drivers were available, but it was a matter of downloading them. And I don't beleive they worked on the 2.6 kernel's I used).
Then the RAID setup required BIOS settings. When you only have remote access to a server (and no KVM-o-IP) that means you need to work through a tech at the DC. Not, umm, ideal.
And finally, there was the issue of 'what if I need to move these disks to a different server'. One that doesn't have the same raid controller. Well, it wouldn't work.
Anyway, I ended up using software raid. I've used it now on a few dozen servers. And I'm really happy with it. Performance seems fine, albeit I'm not using it in really IO critical environments like a dedicated database server. In in 99% of cases I'd now use software raid in preference to hardware raid.
What follows are a few tips I'd like to pass along that may be a help with getting a software raid setup...
If you get the chance setup RAID on / and /boot via your OS installer (on a new system). Doing it afterwards is a real pain [tldp.org].
Build RAID support and RAID1,and RAID5 into the kernel (not as modules). You'll need that if you boot from a raid1 boot partition. Note: if you are using RAID5 you'll need RAID1 built in (since I beleive in the event of a failed disk the raid personaility swaps from RAID5 to RAID1).
With a 2.6 kernel build I've been getting "no raid1 module" errors at the make install phase when building with a RAID-ed / or /boot. The 'fix' is to compile the RAID support you need into the kernel (not as modules) then run: /sbin/mkinitrd -f /boot/initrd-2.6.8.1.img 2.6.8.1 --omit-raid-modules (substituting your kernel image name/version).
Every now and then I've had the kernel spit a drive out a raid array. I've found that sometimes the kernel may be being overly cautious. You can often raidhotremove then raidhotadd it back again. And you may never see a problem again. If you do, it probably really is time to replace the disk.
Rebuilding a RAID array goes smoothly. It happens in the background when the Linux machine is in multi user mode. The md code rebuild guarantees a minimum rebuild rate. From memory it takes about an hour or two to do a 200GB RAID1 array.
You can see the RAID rebuild status in /proc/mdstat. I run a very simple script [rimuhosting.com] to check the RAID status each day and send out an email if it is broken.
If you are using a RAID-ed /boot, grab the latest lilo [rr.com] since IIRC it has better RAID support than what is in the distros I use.
Hard drive-wise I've been happy with Seagate Barracudas. I've had to replace a few failed Western Digital drives. (Just my recommendation from experience, it could just have been good/bad luck on my part).
One neat trick with Software raid is that your drives don't have to be the same size. You do RAID on partitions. And your raid array sizes itself according to the smallest common denominator in the array.
Tip: always create a bit of spare space on any device you are RAID-ing. e.g. a 4GB swap partition. Then if you have a drive fail and it needs to be replaced, and your replacement varies in size slightly you'll still be able to use it. Not all 40/120/200GB drives are created with equal sizes :).
In summary: Software RAID=good. Decent performance. I've had no real kernel bugs with it. No need for BIOS access. Easy to move drives between servers. Easy to monitor failures. Non-intrusive/minimal downtime when recovering a failed devi
software raid5 under Linux (Score:2, Informative)
Overall, I'm very happy with it -- no more rebuilding from scratch every time a boot disk blows!! :) I'd started converting to SCSI boot disks everywhere, but a pair of software RAID2 IDE drives gives me a much better sense of security. My workstation did in fact blow a boot drive a month ago, and rather than being an emergency, I just ambled into Fry's after a week or two and bought a replacement and rebuilt the raid. No muss, no fuss. Feels like living in the third millenium!
I did learn various things the hard way that the HOWTOs don't warn of.
Note that you can't boot off RAID5, only RAID2. The hack they mention of putting /boot on RAID2 and everything else on RAID5 is not worth it with today's drive sizes. Give yourself a 2-16GB RAID2 with a complete bootable system on it, and save yourself mucho grief at very little proportionate cost in disk space.
As of kernel 2.4x, at least, the linux software RAID5 autorecovery is workable but less robust than one might like in the face of serious problems: I had one RAID5 setup totally destroyed because the hardware was flaky leading to constant reboots while RAID5 reconstruction was just underway. After awhile the kernel got confused about the order of the disks (which shouldn't matter, but apparently did) and the whole thing went into a Death Spiral. Lesson: If you're sure the problem is just one flaky disk, feel free to just swap in another and reboot. But if you are in any doubt, play it safe: Switch off RAID autodetect first thing (by fdisk'ing the partition type from FD back to 82. Get the hardware stable, rebuild the RAID by hand, then switch everything back to FD.
RAID5 is a comparative pain in the ass to work with vs RAID2, because under RAID2 any of the partitions can be mounted normally as a non-RAID drive in an emergency, getting you back on the air fast, but not so with RAID5. (You'll want a live Linux CD with a RAID-supporting kernel, likely. Knoppix &tc don't yet ship this way.) So only use RAID5 if the extra space really matters -- the big servers.
BTW: One of the reasons I like software RAID over hardware: If you have hardware RAID and the controller blows and you can't find a matching model, you may be stuck reverse-engineering their RAID scheme to recover your data. No worry about that under software RAID.)
I tested automatic failover to hot spare disks under the kernel, and it worked perfectly for me in a handful of tests. For whatever that's worth.
Do keep an eye on /proc/mdstats readout of your RAID system health. If you're asleep at the wheel and don't notice anything until enough disks fail to bring the whole system down, you haven't gained much. I have a crontab-driven set of Perl scripts which check all sorts of things weekly to minutely and email me if they look wrong: Checking for failed RAID drives is one of the things they do. If you don't have a comprehensive solution like that, the raidtools2 package has an ad-hoc solution specifically to email you on drive failure. USE IT.
FWIW, here's the system I've evolved for partitioning disks in such systems:
Lots of experience...all good (Score:4, Informative)
[root@media root]# more bonnie20.log
Bonnie 1.2: File '/raid/Bonnie.27772', size: 2097152000, volumes: 10
Writing with putc()... done: 14517 kB/s 83.2 %CPU
Rewriting... done: 25060 kB/s 17.1 %CPU
Writing intelligently... done: 41987 kB/s 29.5 %CPU
Reading with getc()... done: 18830 kB/s 96.1 %CPU
Reading intelligently... done: 82754 kB/s 62.2 %CPU
Using an older processor/motherboard is probably not a huge concern. I've used 300 MHz Celerons before. Of course, your performance might not be as high as this, but if you are using this as network attached storage (NFS or SMB), you will likely be limited to 12 MB/sec due to fast ethernet. If you have (and need) gigabit transfer speeds, you should probably use a better motherboard/CPU.
Lastly, remember that you shouldn't skimp on power supplies and an UPS that automatically shuts the system down. The *only* data loss I have ever had on raid5 arrays came because of power-related issues. Heed my warning! 8)
Re:Advice: Get lots of RAM (Score:1, Informative)
D1 D2 D3 D4 D5 D6 P1
where P1 is the XOR of D1 through D6.
If I write to D1, but leave any of D2-D3 alone, then it is necessary to read SOMETHING in order to calculate the new parity. Yes, I know what I'm writing, but unless I overwrite the whole thing, I must perform extra operations in order to update the parity block correctly. These extra operations degrade performance, and are known as the small write problem. As another AC above said, the update can be done with two reads and two writes; read the old D1 and the old P1, then write the new D1, and write P1 to be (old D1 XOR new D1 XOR old P1). It's a bit of trickery, but it does give the correct parity block. It does, however, take two reads and two writes, to update the one block.
Linux (last I looked) doesn't do this. Instead it takes the simpler approach of reading the blocks in the row that it isn't updating (D2 through D6 in this case), and then computing P1 as the XOR of D1 through D6 again.
The small write problem is a big deal. Although the IOs can happen in parallel, the latency for the write becomes the maximum of the reads plus the latency of the parity write. The larger number of IOs also keep the array busy when it could be doing other things, which degrades the performance of those other operations. And it causes this performance degregation for small updates (those under the stripe size), amoung the most common operations. Even if all of your files are big, and written in a streaming manner, the metadata updates are generally in a different row in the array, and are small, isolated writes. A journaled file system, depending on how it is implemented, can be much worse for generating lots of scattered writes. Here [cmu.edu] is a paper from CMU that gives one possible solution (one that isn't implemented by Linux). The traditional solution is write caching--you delay the write until either you've updated the other entries in the row, you've read the other entires in the row, or it is otherwise convienient to do the update (i.e. array isn't busy). This is of course dangerous because your data isn't on disk but in RAM. OTOH, witha good UPS, loosing the contents of RAM is a relatively rare event. To sidestep the volitility of RAM entirely, nice HW raid controllers have some amount of non-volitile memory (either NVRAM or battery backed DRAM) for this purpose. Writeback caching can also help perfomance on non-RAID devices, since it allows you to reorder the writes to minimize head seeks and rotational latency. These two, especially the head seeks, are what make disk IO slow.
You obviously don't have much background in storage. Try reading Chen's classic paper [psu.edu] on RAID, go and search for a few papers the reference it, and then come back and spout off. Until then, quit giving people bad advice.
Re:Lots of experience...all good (Score:3, Informative)
That information may be (and probably is) outdated with regard to SATA. I don't have experience with them yet, though I will be building four 1.75 TB RAID5 (or 1.5 TB RAID6...Linux 2.6 willing) arrays next month that use 250 GB SATA drives.
Comment removed (Score:5, Informative)
Re:Here is a better question (Score:1, Informative)
Re:Advice: Get lots of RAM (Score:3, Informative)
Re:Advice: Get lots of RAM (Score:3, Informative)
a PIII with 128 megs of ram. LVM on top of the array, and I have never run
into a problem with serving files via NFS or SMB. More ram is always nice, of
course, but again, I have not ran into any problems.
SealBeater
Re:Don't go with 3ware (Score:1, Informative)
Buy yourself a good hardware raid card (Score:4, Informative)
Do yourself a favor and get a good hardware raid controller and make sure it has good Linux support. Promise sucks. They advertise Linux support on the box - they lie, only with specific 2.4 kernels. 3Ware has good driver support for Linux included with the Linux kernel source code.
-Aaron
RAIDFrame under OpenBSD works great! (Score:2, Informative)
wd1 at pciide1 channel 0 drive 0:
wd1: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd2 at pciide1 channel 0 drive 1:
wd2: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
wd2(pciide1:0:1): using PIO mode 4, Ultra-DMA mode 5
pciide1: channel 1 configured to native-PCI mode
wd3 at pciide1 channel 1 drive 0:
wd3: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd4 at pciide1 channel 1 drive 1:
wd4: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd3(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5
wd4(pciide1:1:1): using PIO mode 4, Ultra-DMA mode 5
Kernelized RAIDframe activated
raid5 (root): (RAID Level 5) total number of sectors is 960429504 (468959 MB)
The setup is very simple.
# cat
START array
1 4 0
START disks
START layout
32 1 1 5
START queue
fifo 100
It has been quite stable. The box is a 1Ghz AMD with 256MB of ram.
I tried running the same setup under Linux (Gentoo & Slackware). The software RAID would crack under load and fail a disk. I really would give OpenBSD a try with RAIDFrame. You have to recompile the default kernel with RAID suppot but under BSD it is very simple. CVS the source down and:
cd
cp GENERIC RAID
vi RAID and add:
pseudo-device raid 9 # RAIDframe disk driver (Make the number, 9 in this case one more then the number of disc you are using)
option RAID_AUTOCONFIG
save the file and:
config RAID
cd
make clean; make depend; make
cp
cp bsd
and reboot.
Then just read the main pages for raidctl to see how to set it up (hint: look at my raid5.conf above).
Hope this helps.
Re:RAID isn't totally reliable (Score:3, Informative)
I take it you didn't have a drive sitting waiting as a hot-spare?
I got bit by this once. Never again.... now I always have a hotspare waiting to jump into place for an instant rebuild.
raid5 software is great (Score:3, Informative)
also, software raids are hardware independent. they can be modified easily while booted and without rebooting. if a hot-swapable drive is used, downtime can be eliminted by a hot-swap and a rebuild of a failed drive.
also, i have been in a discussion about the new cachefs patch in rescent mm kernel patches(or maybe nitro?), allowing you to use a cache in ram with any filesystem, so you could mount your raid array through the cachefs with a given amount of RAM for write cache
AND, linux software raid works on a per-partition basis, so you can mix and match drive sizes without wasting space. 8 250GB drives can mate up with 4 300GB drives, and then the wasted 200GB can be made into another array.
you can easily add IDE cards and increase the size of your array.
you can spread your array over a large number of IDE cards for better redundancy, no single card will criple your array, and IDE cards are much cheaper than hardware raid cards.
LINUX can be booted from a software raid! while is has trouble on some hardware raids!(driver issues)
i run a software raid5 over 12 seagate 120GB drives with no problems. i get great transfer speeds accross the (gigabit)network and it's easy to manage drive spindown because the system sees each individual drive while hardware raid solutions typically only allow the system to see the array as a single device.
most hardware arrays are mainly configured at boot time. to build or repair an array, your system will not be working. if you run a linux fileserver/firewall, your firewall doesn't function on hardware raid rebuild, while it does in software.
--
though i would go with a faster processor, you should have very good luck, reliability and performance from an 8 device software raid5. and have a nice 1.7TB array
Spend the extra $200 and do it right... (Score:3, Informative)
Spend the extra $200 on a 4 port card... put a *big* fan on the drives because that's the #1 killer and you'll be happy.
Pat
my experiences with software raid (Score:3, Informative)
Anyhow I bought a 3ware 7450 Raid controller and haven't looked back - its brutally fast (over 20-30 megs a second in a sequental write), fully supported in linux and it a piece of cake to setup.
Its not bad at recovering either - I had a power failure and the ups failed later on - machine restarted of course when the power came back on and the 3ware controller automatically rewrote all the parity on the disks - everything was fine. While it wrote the parity the system was up and running instantly (raid was in a fail state of course).
For those looking (Score:3, Informative)
Even better smartd options (Score:4, Informative)
You can get smartd to execute tests automatically, using the -s option.
In my smartd.conf file, I have :
-s (L/../../7/03|S/../.././05)
on the device lines, which means do a weekly online long test at 3 am Sunday, and a daily online short test at 5 am every day.
mdadm running as a daemon, and watching the md arrays is also a good idea.
Fine (Score:3, Informative)
No problems at all. I once had an IDE controller fail - I replaced it (had to reboot of course), and Linux rebuilt the array automagically.
I have not tried using a hot spare.
Warning: a lot of the documentation out there on the web about Linux software RAID is very out of date. If you go this route, DEFINITELY buy the book "Managing RAID on Linux" (O'Reilly). Also be prepared to compile the "raidtools" package, which you need to set up arrays.
I have since added an 8-disk system based on 3Ware's 9000 series SATA RAID controller. I recommend 3Ware for higher-performance systems. (I have 8 250GB disks in a single 1.6TB RAID-5, I get about 180MB/sec read, 90MB/sec write.)
A few other hints (Score:3, Informative)
If you run smartmontools, you can configure smartd to not only monitor the SMART status of the disks, but also execute online tests - have a look at the "-s" option of smartd. For my RAID1 array, for each device, I have -s (L/../../7/03|S/../.././05) entries.
mdadm also has a daemon mode which can monitor the arrays, and if there are any failures, send an email to a designated email address.
Re:Advice: Get lots of RAM (Score:4, Informative)
Probably the best move is to have a cron job examine /proc/mdstat and e-mail you if it's troubled.
RaidWeb.com has nice hardware too. (Score:3, Informative)
The only big disadvantage I experienced at the time was the lack of docs on the serial controller, so I only had the audio buzzer signal to go on when a drive failed. I think the box would have sent a signal over the serial link to the host indicating a failure. Then the host could do something interesting with that signal like send e-mail, call a pager, and so on. It would have been nice to have remote signaling, but in this case I didn't need it. The install site always has someone there to handle taking out the bad drive and plugging in the cold spare.
Re:Advice: Get lots of RAM (Score:3, Informative)
gigabit fiber (Score:3, Informative)
Re:Advice: Get lots of RAM (Score:4, Informative)
Or you can just have mdadmd (pard of the mdadm [freshmeat.net] suite.(comes with my distro (SuSE 9.1))) running, and it'll monitor your raid arrays, and email you when there's a problem.
Re:Experience (Score:5, Informative)
It scares me that they let people like you play with the sort of computing resources that have 50TB of disk space.
Re:Advice: Get lots of RAM (Score:2, Informative)
Everything you need to write is already in RAM except the checksum black. So if you have a 7-drive RAID5 array, the RAID subsystem can take 6 blocks of data, compute a parity block from them then write one block to each drive. It's not like it is going to write random sized chunks of data and can not tell what is going to be written without actually writing it to disk.
Even hardware RAID cards typically don't have a lot of RAM. They also don't write to the drives, re-read what they have just written and create a parity bit from it. Neither does software RAID in Linux.
The only time the system should need to read in order to generate a block is when it is rebuilding after a drive has been replaced.
Heat will be a problem (Score:3, Informative)
8 of those suckers are going to get toasty without plenty of auxilliary cooling.
Re:CONFIGURE IT RIGHT!! small parts... (Score:3, Informative)
If you have one large partition and impending drive failure wipes out any cylindar on that drive, all the data on it is shot. That drive won't be used at all during the rebuild... a rebuild of 250Gb. You are at risk if, during any time of the long rebuild, a 2nd drive fails completely or even coughs up a bad cylindar which can't be redirected.
If you have 6 partitions, only the "damaged" one has to be recovered immediately. Obviously you would want to recover them all as soon as possible since that first drive is probably going to bite the dust soon. Even if you do lose the first drive completely and then a 2nd drive during a rebuild, you at least may not lose everything. Any of the 40 Gb blocks which were rebuilt before the 2nd drive died would have been saved.
Getting a slightly different sized drive for an RMA can also be a problem. What if your original 250Gb drives were actually 250.3Gb and the replacement is 250 Gb even? You aren't going to fit that single 250.3 Gb partition onto the replacment drive. And are you going to call the drive manufacturer and complain that your original drives were too big?
I've had issues with this on hardware RAID. I had to back up 600Gb over the network, wipe the entire array out, rebuild it and restore the data. If it had been software RAID, I could have backed up the data from the last partition into one of the others just to be safe, resized the last one, reformatted and copied the data back.
LVM with multiple partitions would have made it even easier.
Re:Don't screw around - hardware is better. (Score:4, Informative)
Don't forget that hardware raid is a single point of failure. The best solution for the absolute best redundancy and performance is software raid set up to be fault tolerant of controller failures. For example, put two seperate scsi cards in the box, and software mirror your data between them, and then stripe on top of that for added performance if you have the drives. When using striping and mirroring together, always mirror at the lowest level, then stripe on top of that.
The basic idea is:
C == controller
D == disk
R == virtual raid disk
C1 --> D1,D2,D3
C2 --> D4,D5,D6
R1 = mirror(D1,D4)
R2 = mirror(D2,D5)
R3 = mirror(D3,D6)
R4 = stripe(R1,R2,R3)
Re:Stick with hardware RAID (Score:3, Informative)
This is also a good reason to use mirroring rather than fancier schemes like striping or RAID-5, if you can afford the capacity hit. You can always mount the drive individually.
This is a very flawed logic (Score:4, Informative)
This logic doesn't hold. Let's first talk about the performance.
Also, on any reasonably modern system, the software RAID will be faster. You just have a much faster processor to do the RAID processing for you. The added overhead of the RAID5 processing is nothing compared to a 1-2GHz processor.
The actual RAID processing is relatively easy, and any RAID solution, be it hardware or software, that is worth anything will not have any trouble doing the logic (perhaps the cards mentioned are indeed not worth anything). The processing isn't your limiting factor; it is data thoughput. This is where hardware shines. A lot of extra data has to be shipping in and out to maintain and validate the RAID. This can easily saturate busses. A hardware solution allows the computer to communicate only the "real" data between itself and the hardware device, and then allows that device to take the burden of communicating with the individual drives on their own dedicated busses. Sure, that device can become overwhelemed, but I submit to you that if it does, it was poorly designed.
I am not saying that one shouldn't consider software RAID solutions. Just don't consider them because you think the performance will be better.
Now lets talk about data recovery.
I've lost 4 drives out of a 12 drive system at the same time, and Linux has let me piece the RAID back together and I've lost nothing. Was the machine down? Yes. Did I lose data? No. Compare that with a 3ware hardware RAID system where I lost 2 drives. Even thought I probably could have salvaged 99% of the data off that array, the 3ware just would not let me work with that failed array.
Let us be clear: we are talking about RAID5. In RAID5, you simply cannot lose more than one drive without losing data integrity. And it isn't like you can get back some of your files; the destruction will be evenly distributed over your entire logical volume(s) as a function of the striping methodology. So it is quite impractical to recover from this scenario. I don't know what kind of system was being employeed with this 12-drive array that can withstand a 1/3 array loss, but it certainly wasn't a straight RAID5. I can come up with some solutions that would allow such massive failure, but then we aren't comparing apples to apples. I'd be very interested in knowing what the solution was in this example case. It should also be noted that we don't know how many drives were in the system that lost 2 drives, much less what kind of RAID configuration was being used. No conclusion can be derived from the information provuded.
As an aside, more often than not, when we as individuals want a large cheap array, we are less concerned about performance than reliability. We put what we can into the drives, and we hope to maximize our data/$ investment while minimizing our chances for disaster. A software RAID5 is a good solution. Some posts have said that if you can spend so much on the drives, what's stopping you from spending on a nice hardware controller? I submit that perhaps he's broke now! And besides, a controller that can RAID5 8 drives is quite the expensive controller indeed. This has software RAID written all over it.
I bought one of these from Newegg. (Score:3, Informative)
I bought one of these from Newegg. I had a lot of problems with it. I called Silicon Image technical support. They told me that particular chipset did not work correctly, and they would not release working firmware for it.
I told Newegg about this, but they continue to sell them.
Fry's sells them also. I told a Fry's manager that Silicon Image told me they know they don't work correctly. Fry's still sells them.
I would love to find a technically knowledgeable and honest distributor.
My experience (Score:3, Informative)
These drives are all crammed into an old Dell that was my Wintendo a couple of years ago. A few months back, the grilles on the drive-bay coolers I installed got clogged up and I lost one of the drives to overheating. Upon replacing the drive, the rebuild took the better part of an evening (but didn't need to be attended). No lost or corrupt data.
The only major problem I had was that the RAID was dirty in addition to being degraded (insert "your mom" joke here), because I brought my machine down hard before realizing what was going on. In theory, I could have done a raidhotremove on the bogus drive and brought things down normally
I ended up having to do some twiddling to get it to rebuild the dirty+degraded array. I don't remember what that was, but as long as you don't do something boneheaded like ignore kern.log messages about write errors to a specific drive, get annoyed that it's taking so long to cleanly unmount the filesystem, and hard-reset the box, that shouldn't be an issue
RAID-5 data recovery after losing 2 drives (Score:5, Informative)
No, you're not "done period". You'll lose a lot of data, but may still be able to recover some. Likewise when losing one disk in a RAID-0 setup.
Any file that resides entirely outside of the gap in the array can be recovered. How likely that is depends on the details of the filesystem, the striping, and the size of the file (the larger the file, the more likely that a part of it fell into the bit bucket).
Also, not all drive failures are total. You may have a RAID-5 array with one drive that completely failed, and another drive that just has some bad sectors. In that case you should be able to recover most of your data. Or you may have two disks with just a few bad sectors, which is even less bad.
This all depends on being able to force the array to allow access to the device, so that you can mount the filesystem (in read-only mode) and sift through the remains. Some (many? most?) RAID implementations may just give up if two disks in a RAID-5 array (or one disk in a RAID-0 array) are flagged as bad, in which case you really are screwed, even though your data is still there. From what people have been posting here I would guess that Linux SW RAID will let you force it, though I've never needed to try it myself.
Re:CONFIGURE IT RIGHT!! small parts... (Score:3, Informative)
Not quite. In my experience, bad sectors are only remapped by the drive firmware on write. Attempts to read bad sectors will return errors. This makes sense if you think about it; you might be trying to recover data, and the sector might be readable once in a hundred tries, but if you're writing to the sector, then obviously, you don't care about the data that's there already, so it's an opportune time to remap it.
--
Re:Don't screw around - hardware is better. (Score:2, Informative)
http://www.tomshardware.
seems plenty fast to me, guess they fixed the problems, at elast enough to be at the top:)
Re:Please! (Score:3, Informative)
frickin expensive, though... if you need that kind of performance it'd probably be speedier and more cost effective to do a software RAID 0+1
Re:Stick with hardware RAID (Score:2, Informative)