

iSCSI vs. Fibre Channel vs. Direct Attached Disks? 46
mrscott asks: "Does anyone have any good, simple benchmarks about iSCSI performance in a SAN as it relates to fibre channel and direct attached storage? There's a lot of information out there about iSCSI TCP offload adapters that improve performance, but it's still hard to get a handle on even those stats without the original numbers for comparison. We're considering an iSCSI solution from Lefthand networks, but finding independent (and reasonably simple) numbers has proven somewhat difficult, even with the awesome power of Google behind me."
As of right now... (Score:1, Flamebait)
Re:As of right now... (Score:2)
Seriously, I would think with good equipment, it should come resonably close in bandwidth, but iSCSI would probably have a bit more time in response with the protocol overhead. Of course there are benefits to just using 1 network and 1 stack of switches for everything.. (and dangers!)
Re:As of right now... (Score:2)
-psy
Re:As of right now... (Score:5, Informative)
1. Fibre-channel:
Speed = 2Gigabits/sec = 2048 Megabits/sec =~ 256 Megabytes/sec
2. Ultra320 SCSI (direct attached storage)
Speed = 320Megabytes/sec
3. iSCSI (assuming gigabit network link)
Speed = 1Gigabits/sec =~ 100Megabytes/sec
Once 10Gb NICs become common, then iSCSI will have better link speed (doesn't mean it will be faster).
Many things affect the speed of storage systems.
1. raw disk speed
2. raw disk access time
3. interface (iSCSI or Fibre-Channel or UltraSCSI) latency:
iSCSI latency > FC latency > SCSI latency
4. protocol overhead
iSCSI latency > FC latency > SCSI latency
and on and on...
Good yard stick..
If you have five or more 15K drives in a storage system the link speed will be the bottleneck.
Reasoning: six * burst throughput of single drive > link speed.
apples to apples..
1. Local attached storage will generally be faster than fibre-channel or iSCSI as long as the fibre-channel or iSCSI storage system doesn't have some really highend RAID/cache system.
If you have many small hosts, generally throughput should not be an issue except for some hosts, where a highend internal, multi-SCSI-channel RAID controller and SCSI storage systems will be the fastest.
My 0.02cents (taxes extra)
Re:As of right now... (Score:2)
Don't forget that 10G Fiberchannel will probably come at some point as well. And in the much more realistic and cool world, there's also 2.5Gb and 10Gb Infiniband. If the Infiniband guys will just re-engineer the god-awful physical connectors they designed, it could really take hold fast. Plus it was designed as a generic high speed low-latency data transport - you can do storage, IP networking, direct MPI library stuff, etc all through one Infiniband connection. Some of the Infiniband switch vendors ar
Re:As of right now... (Score:1)
Parallel SCSI has higher overhead than FC. Arbitration, selection, and messages (which still use asynchronous transfers) are bandwidth killers. And if pSCSI has lower latency, it is because you are compar
Re:As of right now... (Score:2)
Try more like 10% on average. Why?
a) the link may be full-duplexed but the spindles on the other end are not.
b) Very few applications have sustained bursts of reads and writes. Most have periods of sustained bursts of reads or writes.
Only when one is talking about multiple terabytes does the affect of a) dissappear (if your SAN distributes data across all available spindles).
Re:As of right now... (Score:1)
Of course not, the spindles don't transfer data. Maybe you meant the drive heads are not full duplex, or maybe you mean that the spindles aren't synchronized. The first isn't an issue for the host adapter, which can always maintain a full duplex transfer on its link if it needs it, and for a disk-to-switch link it won't matter anyway because it is a point-to-point connection that won't be saturated by the data rates of a single dis
Performance isn't everything (Score:2, Informative)
Re:Performance isn't everything (Score:2, Insightful)
There are many external SCSI storage systems with integrated RAID and management functions (everything from audible alarms to SNMP/email support). e.g. http://www.promise.com/product/externalstorage.ht m
The cost of disks have fallen so much that the idea of a giga-SAN ($$$$) to master all storage is just plain silly. Local attached external RAID storage with management is all one really needs. Only when talking about multi-Terabytes of data should
Re:Performance isn't everything (Score:2, Insightful)
Re:Performance isn't everything (Score:4, Informative)
3U external RAID storage system.
Holds 14 to 15 disks, fill as you need.
RAID will allow expansion on the fly.
SCSI DISKS: 14 * 146GB = 1.9TB
IDE DISKS: 14 * 400GB = 5.2TB
If you really need to go to the next rack, fibre channel for the link (external fibrechannel RAID storage, no SAN/fc-switch) is still an option.
The only reason I don't like it is there are very few server platforms (apple XServe being the exception) that boot from fibre-channel storage systems. If I need two internal disks (have to be RAIDed and managed) to boot the OS to load the fibre-channel driver to access the external storage why bother?
Most server platforms suck for internal storage and RAID functionality..
1. Sun Sparc:
No HW RAID for internal disks, sw mirroring only. Most model's dont have support for booting from external disks.
2. Apple XServe:
Good SW RAID for internal disks but if a disk fails..
a) backup
b) recreate RAID
c) restore
3. Dell PowerEdge:
Internal HW RAID controllers allow on the fly expansion but controller driver doesn't expose RAID even alerts to OS. For that you need Dell's OpenManagement suite (not support on all OSes).
I prefer external RAID storage, where the storage system provides the management interface OS agnostic.
Re:Performance isn't everything (Score:2)
Additionally, having the boot disks internally (or on a small enclosure) simplifies the bootup of the system which is crucial for problem resolution; this is especially important in
Re:Performance isn't everything (Score:2)
Things have improved with Solaris9 (on V240) and havent played with 10 yet, but still software mirroring is *okay* but really, for the price one pays for a Sun server can't they include hardware RAID (not just mirroring)??
e.g. v240 can house 4 internal disks, why can't there be a integrated hardware RAID controller to do RAID5 on the internal disks?
What you talkin about, Willis? (Score:2)
I have an HP MSA 1000 fibre channel SAN that runs Linux, Netware, Windows 2000 and Windows 2003. The servers connect to the SAN via a fibre channel switch and HP(Emulex) HBAs in the servers. All systems boot from the SAN! There are no disks in any of the servers. This makes replacing failed server (hardware) a 3 minute plug-and-play operation. The hardest p
The only thing I know about iSCSI is ... (Score:2)
Re:The only thing I know about iSCSI is ... (Score:1, Insightful)
Regarding complexity, speed is not an issue as protocol processing is already being offloaded into hardware. The big advantag
Re:The only thing I know about iSCSI is ... (Score:1)
Re:The only thing I know about iSCSI is ... (Score:4, Insightful)
iSCSI is an IP protocol, and it could be running
over anything that sends datagrams. FiDDI, HiPPI,
Myrinet, la la la...
If you dislike iSCSI over Ethernet (and frankly,
it's only interesting in low-performance cases where
IP routing is important for WAN access to NAS,
so I can understand your aversion), don't use it.
But keep iSCSI in your toolkit. The interoperability
and the option to route is extremely valuable.
Re:The only thing I know about iSCSI is ... (Score:2, Informative)
iSCSI works well, and it's as fast as the underlying network (ideally, a direct crossover connection to the target storage, so there's no other traffic to contend with). iSCSI is not as fast as FCAL (2 Gbit/sec max), but only because 10Gbit Ethernet isn't the main course -- yet.
Usually, your storage pool will be Fibre Channel, and some servers connect via FC LUNs (via an FC switch), others will connect
Differences shouldnt matter (Score:4, Informative)
FC and iSCSI are much more expensive than SCSI Ultra320, which is commodity hardware now. FC just sends the data in optic to outside the system, where larger datawarehouses can be managed instead of getting bigger and bigger Unisys boxen.
So if you need terabytes of data all in one place (I mean at least 10 terabytes), consider iSCSI and FC and putting the disks outside the system for better management. We are getting a NAS solution to replace our backup tapes, requirement was 1.2TB. We will get 4x 300GB Maxtor Maxline II SATA disks... the slow cheap ones, and put them in an IBM xSeries 206 which are going at $500 CDN, with an Adaptec RAID card.
Upto 16 SATA 400GB disks can be managed by a simple adaptec raid card, beyond that, think FC arrays.
Re:Differences shouldnt matter (Score:2)
http://www.seagate.com/cda/products/discs
"Formatted Int Transfer Rate (min) 85 MBytes/sec
Formatted Int Transfer Rate (max) 142 MBytes/sec"
5 * 85MB/sec = 425MB/sec
This provides a rough estimate of how much sustained throughput 15K drives have.
Throw in SCSI command overhead and 5 15K drives can saturate a Ultra320 channel.
SATA (Score:1)
So, as I see it, in a power-failure, all that hard work done on ensuring that a modern file-system is always consistent just goes out of the window: meta-data updates happening before the data itself is updated. In fact, that's likely to happen quite a bit with many writes to separate files, since the metadata's slightly mo
As always, the answer is "depends..." (Score:5, Informative)
Seriously, its not so much a speed issue but an issue of how you manage your environment. You can get enough performance out of all three solutions and there are other, more important things to look at.
As you see, I would recommend worrying less about the performance and more about what you really need and what your environment looks like.
If you just want performance for cheap, then local disks are unbeatable. Instead of spending money on expensive fibre or iscsi offload controllers buy tons of cheaper scsi cards, instead of running fibre and buying a fibre switch buy tons of disk drives.
Most people make the mistake to worry about capacity - in reality, its the number of spindles you need to look at and the capacity or your storage is a result of that. Figure, each disk attached gives you about 100 i/o ops per second. If you need to do 5000 ops, you need 50 disks - no matter how they are connected. Figure a disk can get you 20 MB/sec - if you need 200MB/sec throughput, thats 10 disks.
In that example, I need 50 disks then to satisfy my requirements. Next, take the max throughput and divide it by 3 - that's 66MB/sec for fibre then, 100MB for scsi and 33MB for iscsi over gigabit. So to run my 200MB/sec I'd need only 2 scsi channels, 3 fibre or 6 iscsi connections.
Next of course can't have 50 drives on 2 channels - more than 3 disks drop the transfer rate dramatically... Since fibre and iscsi mask the physical spindles they don't care but I need to have 16 scsi controllers to really run the disks.
Peter.
Re:As always, the answer is "depends..." (Score:3, Informative)
Re:As always, the answer is "depends..." (Score:2)
Have you estimated a $$$ cost for this wasted disk space?
Even if you factor whatever extra cost you want to associate managing DASD vs a SAN, do these extra costs justify the cost of a SAN?
Ok iSCSI *might* be cheaper but does it improve the way you manage storage or does it just abstract the problem away from your host OS(es)?
Re:As always, the answer is "depends..." (Score:1)
It sounds like your environment and requirements might be a good candidate for a Network Appliance filer... NetApp pioneered iSCSI, and you'd also get the benefit of instant snapshot backups, the fact that they already have (working) MPIO drivers for Windows, the performance of a RAID4 Fibre Channel storage system, as well as enterprise-level clustering/backup/recovery and a host of other bullet items you can find on your own.
At risk of sounding like a dreaded sales person, expand your se
Re:As always, the answer is "depends..." (Score:2)
ostiguy
The acid test: (Score:5, Informative)
The acid test -- for me anyway -- is seeing LARGE customers (banks, airlines, government agencies [pick a 3-letter acronym], pharmaceuticals, major industries such as oil and energy, entertainment companies and movie studios, etc.) implementing iSCSI on an equally LARGE scale, and quite successfully.
With few exceptions, if the underlying Ethernet network is functioning properly, iSCSI performs remarkably fast. No, you won't get the 2 Gbit FCAL rates -- *yet* -- but we have customers running dozens of large (>TB) databases off a single appliance over one or two GigE ports and iSCSI.
Generally, it's recommended to segment off the iSCSI traffic so it's not routed or mixed with public traffic anyway, but even those (small) customers that pipe all of their storage appliance data through a single 10/100 Ethernet interface only have problems if they put too many users on there as well. (A direct crossover cable is *ideal* for iSCSI.)
In addition, the Microsoft iSCSI initiator has finally outgrown its initial bugs/problems (with our help in some cases), and is darn solid with plenty of different targets.
I'd love to drop a bunch of example company names, but I'm sure those companies consider that information to be competitive, and it's not my place to divulge it. Any large company you can think of already has an investment in FC-AL, and all but a very small percentage have iSCSI infrastructures as well. Medium-size and small (50 employees) companies are also seeing HUGE benefits from iSCSI their own implementations.
The 0-day acid test (which works amazingly well in our labs with the right HBAs) is SAN booting over iSCSI. Imagine having an nnn-Terabyte set of storage, from which ALL your servers boot EVERYTHING. Not a single magnetic disk is required in the servers themselves. Makes server clustering and blade/grid computing so very attainable
(As an engineer for a major storage vendor (FC/iSCSI/near-line IDE storage/archiving), I work with all of this stuff on a daily basis. Not saying I'm an expert, just that I kinda know what I'm spouting here...)
Re:The acid test: (Score:2, Flamebait)
That's called, "having all your eggs in 1 basket", and we all know what a bad idea that is...
Re:The acid test: (Score:1)
Having them all in one place makes mirroring the entire data set in real time more feasible. Heck, you can even mirror across a WAN link and have an offsite DR location that's always in sync with the original nnn TB data set.
Having all the disks in one place can also allow for more security. Admins may have access to the servers, but not the media itself, so there's diminished risk in an entire hard disk going "missing," a la NASA JPL and/or Los Alamos. If you guard that, plus don't all
Re:The acid test: (Score:1)
Re:The acid test: (Score:2)
Don't pull shit out your ass. I'm a DBA, with a BSci in Comp. Sci.
We have a Hitatchi SAN, and one day, guess what? The SAN crashed!! All systems using it were down for a day: AS/400s, HP PA-RISC boxen, large Alpha VMS servers, and mainframes.
So fuck you and the horse you rode in on.
Re:The acid test: (Score:3, Insightful)
That's called, "having all your eggs in 1 basket", and we all know what a bad idea that is...
About the booting part: at work we boot from local disks because we Unix SAs don't have control of the EMCs, thus if a machine does not boot, we cannot do much beside calling someone who has no idea about Sparc machines. If we Unix SAs were able to control the EMCs and everything related like the FC switches, then we would boot directly from SAN. If I were able to control the iSCSI storage box, the routers and s
the baskets aren't equal though... (Score:2)
I could have a lot of fun with that metaphor but i'm sure you get the point without it.
best speed to clusterf*ck... (Score:4, Funny)
Mike
Re:best speed to clusterf*ck... (Score:2, Informative)
iSCSI LUNs can be pretty friggin' huge. More than once, I've seen a customer nuke the holy bejeebers out of a 950GB iSCSI LUN all at once, with a single click (the equivalent of "LUN DESTROY"). I actually had a WebEx session to a customer once and watched him delete a 1.3TB LUN before I could stop him.
If you're on plain JBOD, you have to get out your restore tapes and pray to the tape gods. If you're on a SAN/NAS appliance (like NetApp), you can usually restore that data in a matter of seconds
Spindle transfer rates (Score:4, Informative)
My experience is with very high transaction volume OLTP databases (oracle) backing a financial website. I've found that neither SCSI nor FCAL adapters limit performance significantly. This was with qlogic qla3200 adapters, or with highend adaptec Ultra320, on Solaris 9 and the last few versions if RedHat enterprise. Only the older versions of redhat had some kind of problem with the qlogic driver, plus bounce buffer IO, which drove down performance. But then to be nit picky, that was the driver not the HBA. Solaris was always fine, and now redhat is too.
The main performance challenge was *always* tuning the database and spreading out on lots of spindles. The HBAs at over 200M/sec each never posed a problem on larger sun boxes (8 or more procs) with 7 or 8 way parallel sequential reads going. On smaller hosts or smaller disk arrays,k the problem was always on the host itself or the disk seek times respectively, not the hbas themselves.
A 10k rpm drive will do about 70 mbytes/sec off the outer platter (near block 0) and as a rule of thumb, a 2Gbit fcal adapter will do 200mbytes a second (at least on solaris or newer redhat EL). So my dual qlogics would do 400Mbytes a second under absolute optimal disk access, but typically it's not that perfect 8 way parallel *serial* scan off the outer platter, its usually farily random.
So in the high end database applications (datawareyouse or OLTP) least the usual tuning challenge (and $$ for that matter) are with getting a fat spread across a lot of spindles, and making sure the application is either caching well (OLTP) or doing orderly, sequential scans (datawarehouse)
Bad ?. How about what do you want to do with it?? (Score:2, Informative)
Ummm What are you doing with it? What's your budget, and what are your expectations? Some people think a TB is big, some consider 20+ TB a building block standard (I do).
1. Look on the net and in mags. - read read read
Is this a big implementation, or a small one. I'd bet a small one, or you probably wouldn't be asking here - that probably means you don't want iscsi. Direct FCAL on multi-port RAIDs might be your speed - if you aren't big en
Reverse iSCSI? (Score:2)
I've seen plenty of adapters that you can hook to a SCSI device and call it an iSCSI device but that assumes you have a proper iSCSI HBA.
What I need is something I can attach to a SCSI HBA that has an ethernet connector on the other side, so I can connect it to a network and do iSCSI over it.
This particular instance is for an old machine that noone's ever going to make an iSCSI HBA for to connect it to a tape dri