Hardware For Bulk IDE Hard Drive Burn-In? 51
r0gue_ asks: "I work for a mid-size OEM hardware manufacturer. We ship approximately 300 to 500 IDE HDs every month across all our units. Currently we experience about a 4% failure rate (Maxtor and WDs), though in recent months it has been a couple percent higher. The problem is our systems are dedicated boxes with a non end-user friendly form factor. Virtually every physical HD failure results in an RMA. What we are looking for is a hardware based IDE HD burn-in platform. Something that we could drop a dozen or so drives in at once, stress test them for a day or two, then put them into inventory for builds. I know the HD manufacturers and larger OEMs use them but I have not been able to track down anywhere we could purchase one. Right now moving to SCSI or a form factor that supports externally removable drives is not an option. I was hoping that the Slashdot community could point me in the right direction."
How-to (Score:4, Informative)
write bash script that checks dmesg for how many drives are in the system and invoked the follwing perl script for each drive.
Write perl script that does this.
formats and partitions drive to max size,
copies a kernel or some other large file onto the disk until it is full.
monitors syslog for IDE errors
md5sums the files to make sure they all match.
reports an error if the MD5 doesnt match.
unless you get hotswap controllers you will have to reboot everytime you want to test another batch of drives.
if you dont wish to write this perl script i can be hired to do it for you.
Re:How-to (Score:3, Informative)
Comment removed (Score:5, Informative)
Western Digital and Maxtor are equal? (Score:5, Informative)
Slashdotters! If you don't find a story interesting, please don't complain and call Slashdot lame. Just ignore the story. Do you complain to your local newspaper that they should not publish recipes because you don't cook?
Comment about the Slashdot question: The wording of the question seems to imply that you believe that Maxtor and Western Digital hard drives have an equal failure rate. That has not been my experience. My experience has been that Western Digital are the most reliable hard drives. I'm very interested to know the experience of other readers.
Western Digital went through a bad stretch in which they experienced a problem that caused high failure rates several years ago, but that was cured.
It's shocking that you are in the computer business and knowingly shipping products with a 4% failure rate. That's very expensive and annoys the customers.
However, you are on the right track. Electronic products have what is called "infant failure". Most failures occur in the first week. During 192 hours (one week), the failure rate falls typically by a factor of 100 or even 10,000. At the end of one week most failures have already happened.
It's very easy to write a program that exercises a hard drive. Just copy files back and forth from folder to folder. It is easy to write a program that fills a hard drive with files, then erases them and starts again.
The Promise Ultra133 TX2 [promise.com] supports adding four more hard drives to the 4 already supported by modern motherboards. Eight is enough for one test computer, usually, because the power supply won't support more. Be careful to use delayed start. Maybe you will need more powerful power supplies than you normally use.
Make SURE that you are not having troubles with heat. Are your drives cool when they are installed in your product? High heat will cause high failure rate.
Re:How-to (Score:3, Informative)
Now, my motherboard supports PCI 64 at 66Mhz, with a bandwidth of 532MB/s, this would give 8.3MB/s per disk. Still not a lot, and you'd have to find a PCI 64 Firewire card with a lot of connectors, because at least my motherboard has only two slots.
My 80GB disk can do 40MB/s quite easily according to hdparm, so with my available bandwidth I could support about 13 drives, let's say 10 to compensate for overhead and other things on the bus. I think that 40MB/s is quite near the limit of Firewire, so I might need ATA instead. With two serial ATA cards with at least 5 connectors on each I suppose it'd be possible. Parallel ATA would also work, I guess, but the wiring would be really complicated with so many drives, especially because you want to have a drive per cable for maximum performance.
Re:How-to (Score:5, Informative)
For example, if it's just bad spots, then you'll want to do as many reads and writes as possible. For that, the fastest thing would be a little C program that reads and writes different patterns to the raw device linearly.
On the other hand, if the failures are tied to seeks, you'll want to write to semi-random locations on the device, to force maximum seeks. Or if you see a mix of both, then your best bet might be to follow m0rph3us0's plan, perhaps tweaking it a bit to better simulate normal filesystem efficiency (and you can just do bit compares rather than md5sums if CPU is an issue).
You should also keep an eye on heat issues. The burn-in should happen at temperatures that are like what they will be in the end systems. If you pack 8 seeking drives into some cases, they'll cook. If you leave them in the open air, they might not trigger the failures you are seeing in the field. Try to match measured operating case temperature.
Oh, and don't forget to measure whether this burn-in is really helping. Take stats now, and keep tracking causes of return. It could be that the drives are sensitive to noisy power or vibration or something else that your burn-in won't catch.
Case solution (Score:3, Informative)
We built some disk arrays using a front-loading IDE case with drive trays. This one is pretty pricey but it's _nice_ hardware:
https://www.rackmountplus.com/spec.asp?ID=RMAC4D -IDE
That, plus a couple RAID cards (like 3ware's new 12-port cards) in a 64/66 PCI slot and bonnie++ would do a pretty good job of burning in your drives. You could flip drives in and out in a few seconds.
Re:Here's a solution: (Score:4, Informative)
StorageReview [storagereview.com] has a Drive Reliability Survey that lists statistics for many drive families. For example, WD 205Bx drives are near the top of the rankings (99th percentile) while the 600Ax is near the bottom (10th percentile).
Re:Western Digital and Maxtor are equal? (Score:2, Informative)
As far as powering them up in a sequence, there is no need to do that really, you can just turn on all the power supplies with the same switch. That's a little trickier to do with ATX, but cyberguys sells a adapter to make an ATX power supply act like an AT one with an external switch, and AT style motherboard connector. Or, since you aren't using the motherboard connector, you could just send the power supply the on signal the same way the motherboard does (read the ATX spec).
Most redundant power supply systems are really parallel power supply systems, so if you really need something like 800 watts, just look for a case that can take redundant supplies and put two 400s in, or two 480s, or whatever you can afford.
Re:3Ware card + Removable Hard Drive Bays (Score:3, Informative)
Re:Here's a solution: (Score:3, Informative)
My main reason for suggesting Samsung, aside from the joys of a real 3 year warranty and the fact that Samsung drives really are value-priced, is that my return rate, and the return rates of several other resellers I know, has been exceedingly low.
Re:How-to (Score:2, Informative)
Can your PC really do sustained writes to >8 drives without getting into performance issues?
Actually, speed is a good point. I guess I hadn't thought about that so much as part of the setup, but I guess if you're exercising disks to make them fail, you want to do it as fast as possible... even though, in your end-system where the disks will be used, a Firewire stress-test may be more realistic in terms of disk behaviour.
Either way, there's tradeoffs.