High Sustained HD Transfer Rates on a Budget? 18
aibrahim asks: " I need to be able to sustain at least a 23MB/s (That is MEGABYTES per second) transfer rate over the course of about 2 hours. I'd like to get 60MB/s sustained. I also have to be able to perform seeks quickly because I am going to need random access. Can ATA RAID arrays really do it? What would be the difference between using ATA/66 and ATA/100 arrays? What other budget conscious ways are there to get it done ? How about sharing this fast storage across a network? For some context, the application is non-linear editing of uncompressed standard definition television, multiple streams if possible."
You're asking a lot, but it can be done. (Score:1)
I have the Adaptec IDE raid card (AAA-UDMA) with two IBM 37.5 gig hard drives in a stripe set, and it can sustain 27 MB/s read. (It contains footage for a DV feature; it's about 80% full.)
The new IBM 75 gig IDE drives are very fast; you might look at those.
The total amount of data you are looking at is about 170 gig by my calculations. So you'll need three of the IBM 75 gig drives, plus the adaptec card (which can support up to 4.) That'll give you 225 gig at a cost of about $1600. Or you could try to use WinNT/Win2k software striping. This setup might even get you up near saturating the PCI bus, at about 90MB/s.
Avoid Promise cards at all costs; I have lost a lot of time and data trying to use them.
You might check out Storage Review [storagereview.com] as a good place for disk info.
If you're really on a budget, you shouldn't be using uncompressed video. It's much harder to work with, eats a lot more storage, and isn't any better looking to 99% of the population than DV.
Marshall Spight
Re:You'll probably... (Score:1)
If the original seeker really, really does mean "random access", then they're shit outta luck. Our servers where I work use the hottest, greatest expensive Cheetah SCSI drives for extremely random-access reads and writes. We're bottlenecked on the rate at which the drive arm can bounce around on the platter. During peak activity, we're only seeing about 10MB/s. To get any better performance, you'll either need to restructure your database such that you are seeing sequential access more often, or you're going to have to spend some major bucks on a SAN product or buy enough RAM to cache up a sizeable fraction of your read/write requests.
-- Guges --
Re:You'll probably... (Score:1)
Running more than one stream off the same set of disks will cause the access patterns to go to crap & the throughput to go out the window. Doing multiple streams will probably need a stripe set per stream.
Editing compressed video streams is fairly I/O intensive. Trying to deal with uncompressed streams is *not* going to be cheap.
i did it. (Score:1)
config :
Sun A5200 fibre channel disk array
18GB x 8 FC drives.
Sun A5000 series machine with 8 x 450MHz ultrasparc-iis, 2 x 256MB cache FCAL cards.
4 x 1 GB ethernet cards (1000BT).
cost : if you have to ask.....you cant afford it.
Oops... (Score:1)
--
Re:Fibre channel (Score:1)
Too bad, huh?
--
Don't forget the RAM (Score:1)
I know cost is an issue, so I say all this assuming you don't want to buy an expensive caching RAID adapter.
Raid descriptions (Score:1)
Remember all RAIDs are not created equal.
23 MB/s == a LOT of bandwidth (Score:1)
23 megs a second is a lot of bandwidth, whether we're dealing with a network or not. I'm curious as to what the intended use of this will be.
=================================
Perc3DI RAID cards (Score:1)
remember the saying... (Score:2)
If you go with IDE (and software RAID) you've got cheap and fast down.
SCSI RAID would, of course, be "fast" and "good".
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
A traditional PCI mainboard is a bottleneck! (Score:2)
First off, AVOID:
Additionally, what you need to consider:
-- Bryan "TheBS" Smith
Even Faster access, 4Gbit! (Score:2)
Fibre channel (Score:2)
I remember checking out the prices at pricewatch for 9.1 gb, 7200RPM FC drives -- I don't know how they are so cheap, but some their cost less than $100 with shipping.
When you starting talking about HBAs (Host bus adapters) the money comes in
driver support is also a problem... I don't know if linux currently has any good HBA drivers yet...
Damn, this is an incoherent post. Main point: FC is incredible and incredibly expensive.
willis.
Check out Medea... (Score:2)
bakes
--
IDE is faster than we think (Score:3)
One lovely alternative is using a UDMA/66 hardware RAID, suck as the Promise RAID controller. Throw in four drives @ RAID 0 and I'm sure you'll get (at least) 35MB/sec sustained.
Don't forget that 60MB/sec sustained is half the PCI bus theoretical bandwidth, so probably a 64-bit PCI solution would be best.
What about the rest of your hardware? (Score:4)
In theory a normal PCI bus can reach 132MB/s. However you not only are reading (writing?) that data from your harddrive, I'm assuming that you also need to put that onto your display.
Don't forget cache issues, you DMA that into memory, then read it out to the processor. Can your memory handle that kind of access? Your putting a lot of stress on the memory bus. If your main code doesn't fit into the processor cache (or isn't optomised to fit well) Sure the lastest gigahertz CPUs can deal with the data just fine, but typically PCs can't keep up with the data flow.
For fast disks, SCSI rules. while ATA now allows taged queueing, AFAIK nothing impliments taged queueing in ATA disks, while scsi does this as a matter of course. Meaning that you will want to select disks based on that feature.
Remember, your application is time critical. If a frame is late it matters.
Now can ATA disks keep up? I don't know. Are scsi disks going to be better? Probably. Is the difference enough to matter? Maybe.
In any case, no single disk can keep up with your requirements. What you need is a raid 0+1 so that data can always be read from two disks, in a good implimentation you read from which ever drive is less busy now. Unfortunatly your writing costs go up as you add more drive to make the reading faster. If you can put data on a different disk so that you never read from the same one you write to you will have better luck.
You'll probably... (Score:4)
With 'older' UW SCSI hardware (2 1997/1998 9GB 7200RPM IBM drives) I can sustain ~12MB/s from each, and if I add in my 10krpm drive, I can sustain a total that essentially maxes out my 40MB/s UW SCSI link. If you *need* to keep near 60MB/s, U2W is really your only cost effective choice. Get 4 drives and a card... yeah, it'll run you a little $$$, but you actually will have the performace (striping the disk set, of course).
If you have a dedicated 100BaseT Ethernet link, you might be able to get 20MB/s but not 60... certainly not onto the same system as the drives (PCI 32b/33MHz is ~132MB/s max).
Best of luck.
--