Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
Data Storage Hardware

iSCSI vs. Fibre Channel vs. Direct Attached Disks? 46

mrscott asks: "Does anyone have any good, simple benchmarks about iSCSI performance in a SAN as it relates to fibre channel and direct attached storage? There's a lot of information out there about iSCSI TCP offload adapters that improve performance, but it's still hard to get a handle on even those stats without the original numbers for comparison. We're considering an iSCSI solution from Lefthand networks, but finding independent (and reasonably simple) numbers has proven somewhat difficult, even with the awesome power of Google behind me."
This discussion has been archived. No new comments can be posted.

iSCSI vs. Fibre Channel vs. Direct Attached Disks?

Comments Filter:
  • by virid ( 34014 )
    in terms of speed, iSCSI can't touch 2GB Fibre-Channel...
    • Especially the speed at which fiber channel blows the department budgets!
      Seriously, I would think with good equipment, it should come resonably close in bandwidth, but iSCSI would probably have a bit more time in response with the protocol overhead. Of course there are benefits to just using 1 network and 1 stack of switches for everything.. (and dangers!)
    • by Xross_Ied ( 224893 ) on Thursday December 30, 2004 @08:33PM (#11223853) Homepage
      Umm not exactly..
      1. Fibre-channel:
      Speed = 2Gigabits/sec = 2048 Megabits/sec =~ 256 Megabytes/sec

      2. Ultra320 SCSI (direct attached storage)
      Speed = 320Megabytes/sec

      3. iSCSI (assuming gigabit network link)
      Speed = 1Gigabits/sec =~ 100Megabytes/sec

      Once 10Gb NICs become common, then iSCSI will have better link speed (doesn't mean it will be faster).

      Many things affect the speed of storage systems.
      1. raw disk speed
      2. raw disk access time
      3. interface (iSCSI or Fibre-Channel or UltraSCSI) latency:
      iSCSI latency > FC latency > SCSI latency
      4. protocol overhead
      iSCSI latency > FC latency > SCSI latency
      and on and on...

      Good yard stick..
      If you have five or more 15K drives in a storage system the link speed will be the bottleneck.
      Reasoning: six * burst throughput of single drive > link speed.

      apples to apples..
      1. Local attached storage will generally be faster than fibre-channel or iSCSI as long as the fibre-channel or iSCSI storage system doesn't have some really highend RAID/cache system.

      If you have many small hosts, generally throughput should not be an issue except for some hosts, where a highend internal, multi-SCSI-channel RAID controller and SCSI storage systems will be the fastest.

      My 0.02cents (taxes extra)

      • Don't forget that 10G Fiberchannel will probably come at some point as well. And in the much more realistic and cool world, there's also 2.5Gb and 10Gb Infiniband. If the Infiniband guys will just re-engineer the god-awful physical connectors they designed, it could really take hold fast. Plus it was designed as a generic high speed low-latency data transport - you can do storage, IP networking, direct MPI library stuff, etc all through one Infiniband connection. Some of the Infiniband switch vendors ar
      • Actually, the FC rate is in multiples of 1.0625 Gb/s, so it is 2.125 Gb/s for 2GFC, but you need to divide by 10 instead of 8 to get the symbol rate, so 2GFC delivers 212.5 GB/s. Unlike parallel SCSI, though, Fibre Channel is full duplex, so with a good mix of reads and writes FC will move around 400 GB/s.

        Parallel SCSI has higher overhead than FC. Arbitration, selection, and messages (which still use asynchronous transfers) are bandwidth killers. And if pSCSI has lower latency, it is because you are compar
        • Unlike parallel SCSI, though, Fibre Channel is full duplex, so with a good mix of reads and writes FC will move around 400 GB/s.

          Try more like 10% on average. Why?
          a) the link may be full-duplexed but the spindles on the other end are not.
          b) Very few applications have sustained bursts of reads and writes. Most have periods of sustained bursts of reads or writes.

          Only when one is talking about multiple terabytes does the affect of a) dissappear (if your SAN distributes data across all available spindles).

          • a) the link may be full-duplexed but the spindles on the other end are not.

            Of course not, the spindles don't transfer data. Maybe you meant the drive heads are not full duplex, or maybe you mean that the spindles aren't synchronized. The first isn't an issue for the host adapter, which can always maintain a full duplex transfer on its link if it needs it, and for a disk-to-switch link it won't matter anyway because it is a point-to-point connection that won't be saturated by the data rates of a single dis
  • You want manageability. You want the ability to take "some disk" and add it to a server, anywhere, at any time. You want the ability to grow/shrink the filesystems on those servers. You want redundancy, and you want top notch vendor support. Direct disk might be faster, with local processing handling the FS buffering in local RAM, but what happens when ServerA needs 20G of the 100G disk you installed in ServerB?
    • With disks so cheap just add another disk to ServerA.

      There are many external SCSI storage systems with integrated RAID and management functions (everything from audible alarms to SNMP/email support). e.g. http://www.promise.com/product/externalstorage.ht m

      The cost of disks have fallen so much that the idea of a giga-SAN ($$$$) to master all storage is just plain silly. Local attached external RAID storage with management is all one really needs. Only when talking about multi-Terabytes of data should
      • And when you've filled the rack that the server is in, where do you stick your disk array? Or do you only populate your racks 1/3 full, to allow for additional capacity, just in case? When the server in the 1U case needs more disk, where do you add it? How do you add space w/o taking the server down? A TB is 1000 GB, which is 100 10GB servers. Any decent sized shop will easily suck up a TB. Any large shop will devour lots more.
        • by Xross_Ied ( 224893 ) on Thursday December 30, 2004 @09:47PM (#11224348) Homepage
          1U server:
          3U external RAID storage system.
          Holds 14 to 15 disks, fill as you need.
          RAID will allow expansion on the fly.
          SCSI DISKS: 14 * 146GB = 1.9TB
          IDE DISKS: 14 * 400GB = 5.2TB

          If you really need to go to the next rack, fibre channel for the link (external fibrechannel RAID storage, no SAN/fc-switch) is still an option.

          The only reason I don't like it is there are very few server platforms (apple XServe being the exception) that boot from fibre-channel storage systems. If I need two internal disks (have to be RAIDed and managed) to boot the OS to load the fibre-channel driver to access the external storage why bother?

          Most server platforms suck for internal storage and RAID functionality..
          1. Sun Sparc:
          No HW RAID for internal disks, sw mirroring only. Most model's dont have support for booting from external disks.

          2. Apple XServe:
          Good SW RAID for internal disks but if a disk fails..
          a) backup
          b) recreate RAID
          c) restore

          3. Dell PowerEdge:
          Internal HW RAID controllers allow on the fly expansion but controller driver doesn't expose RAID even alerts to OS. For that you need Dell's OpenManagement suite (not support on all OSes).

          I prefer external RAID storage, where the storage system provides the management interface OS agnostic.
          • Sun v440s have hardware mirroring on internal disks, although it's only supported on 2 of the 4 (i.e. you can mirror any 2 disks using the hardware RAID, after that you need to use software RAID). In any event, I've not seen any issues in using software mirroring on any server (Sun, HP or IBM). The extra CPU load is minimal.

            Additionally, having the boot disks internally (or on a small enclosure) simplifies the bootup of the system which is crucial for problem resolution; this is especially important in

            • Perhaps my opinion is coloured by memories of Solaris7 (on a 420R), where one had to have veritas filesystem to get decent journaling and mirroring.

              Things have improved with Solaris9 (on V240) and havent played with 10 yet, but still software mirroring is *okay* but really, for the price one pays for a Sun server can't they include hardware RAID (not just mirroring)??

              e.g. v240 can house 4 internal disks, why can't there be a integrated hardware RAID controller to do RAID5 on the internal disks?
          • The only reason I don't like it is there are very few server platforms (apple XServe being the exception) that boot from fibre-channel storage systems.

            I have an HP MSA 1000 fibre channel SAN that runs Linux, Netware, Windows 2000 and Windows 2003. The servers connect to the SAN via a fibre channel switch and HP(Emulex) HBAs in the servers. All systems boot from the SAN! There are no disks in any of the servers. This makes replacing failed server (hardware) a 3 minute plug-and-play operation. The hardest p
  • ... avoid it at all costs. Everyone I've talked with about iSCSI, from driver writers to end users does not like it. ATA over ethernet shows more promise, as it's much more simple than iSCSI. We'll see what future will bring ... but for now I'd stick with fibrechannel.
    • by Anonymous Coward
      What do you mean by 'more promise'? I agree that iSCSI is complicated, but there is a reason it was done at the IP level. Another thing that diffrentiates iSCSI from ATAOE or even HyperSCSI (SCSI over Eth), is its inbuilt catering for redundancy,scalability and error management. In fact, if you look at the iSCSI spec approx 40% caters to error management, and in IMHO is done well.

      Regarding complexity, speed is not an issue as protocol processing is already being offloaded into hardware. The big advantag
    • The problem with iSCSI is the Ethernet component. It's too slow, craps all over all other Ethernet traffic and nowhere near as reliable as fibrechannel.
      • by aminorex ( 141494 ) on Thursday December 30, 2004 @10:49PM (#11224742) Homepage Journal
        but, but... iSCSI has nothing to do with Ethernet.
        iSCSI is an IP protocol, and it could be running
        over anything that sends datagrams. FiDDI, HiPPI,
        Myrinet, la la la...

        If you dislike iSCSI over Ethernet (and frankly,
        it's only interesting in low-performance cases where
        IP routing is important for WAN access to NAS,
        so I can understand your aversion), don't use it.
        But keep iSCSI in your toolkit. The interoperability
        and the option to route is extremely valuable.
      • No avoidance is necessary, and unless you're familiar with iSCSI, your "evidence" is anecdotal at best.

        iSCSI works well, and it's as fast as the underlying network (ideally, a direct crossover connection to the target storage, so there's no other traffic to contend with). iSCSI is not as fast as FCAL (2 Gbit/sec max), but only because 10Gbit Ethernet isn't the main course -- yet.

        Usually, your storage pool will be Fibre Channel, and some servers connect via FC LUNs (via an FC switch), others will connect
  • by mnmn ( 145599 ) on Thursday December 30, 2004 @09:43PM (#11224323) Homepage
    Both iSCSI and FC are networked version of SCSI, and all 3 technologies are much faster than their respective disks, thereby not being the bottleneck at all. After Ultra160, the standard PCI channel is saturated, and 64-bit PCI like PCIX is needed for Ultra320, all the while usually even in the burst mode ( from cache) disks cant saturate this available bandwidth, say 6x RAID5 15K RPM disks in read mode.

    FC and iSCSI are much more expensive than SCSI Ultra320, which is commodity hardware now. FC just sends the data in optic to outside the system, where larger datawarehouses can be managed instead of getting bigger and bigger Unisys boxen.

    So if you need terabytes of data all in one place (I mean at least 10 terabytes), consider iSCSI and FC and putting the disks outside the system for better management. We are getting a NAS solution to replace our backup tapes, requirement was 1.2TB. We will get 4x 300GB Maxtor Maxline II SATA disks... the slow cheap ones, and put them in an IBM xSeries 206 which are going at $500 CDN, with an Adaptec RAID card.

    Upto 16 SATA 400GB disks can be managed by a simple adaptec raid card, beyond that, think FC arrays.
    • See...
      http://www.seagate.com/cda/products/discsa les/ente rprise/tech/0,1084,656,00.html

      "Formatted Int Transfer Rate (min) 85 MBytes/sec
      Formatted Int Transfer Rate (max) 142 MBytes/sec"

      5 * 85MB/sec = 425MB/sec

      This provides a rough estimate of how much sustained throughput 15K drives have.
      Throw in SCSI command overhead and 5 15K drives can saturate a Ultra320 channel.
    • by Hydraq ( 706666 )
      SATA scares me a little, simply because Native Command Queuing doesn't allow the OS to impose any kind of ordering constraints on the commands, the way that Tagged Command Queuing does.

      So, as I see it, in a power-failure, all that hard work done on ensuring that a modern file-system is always consistent just goes out of the window: meta-data updates happening before the data itself is updated. In fact, that's likely to happen quite a bit with many writes to separate files, since the metadata's slightly mo
  • by loony ( 37622 ) on Thursday December 30, 2004 @10:33PM (#11224624)
    Unfortunately you omitted to say what size of installation you're talking about... I work at a large company and layout and maintain a development shop for like 600 developers... I prefer direct attached disks for a simple reason - if something fails, I have less people complaining about it :-)

    Seriously, its not so much a speed issue but an issue of how you manage your environment. You can get enough performance out of all three solutions and there are other, more important things to look at.
    • Price: direct attached storage being the cheapest usually, then iscsi being a little more and fibre being by far the most expensive.
    • Redunancy: the higher the price, the more redundancy you usually get - its not a technology issue but a cost issue again... dasd is usually at the same level as iscsi since most manufacturers just give you a preconfigured computer with dasd and then sell it as iscsi... Fibre gives you the highest level of redundancy - because if you already go spend 5 million on symetrix frames its easy to justify the additional $400K for one more smaller connectrix. If you only talk about $4000 disk trays its hard to justify having an electrician comming in to give you redunant power for another $3000...
    • Clustering is another reason to go with fibre or iscsi - direct attached storage is usually a bad idea there...
    • Backups are another consideration.. Backup through ethernet is much slower than fibre tape units through a san...
    • Fibre and iscsi are not dedicated. So if I have a server with direct attached disks that happily does 2000 io/sec today, it will do so tomorrow unless that server has issues. If I have fibre or iscsi and lets say I run year end reports on another box, that heavy load there can drop my io rate to 20% or even lower.
    • Fibre is a bitch to configure the first time. direct attached disks are easy and iscsi is usually managable - but getting fibre right the first time is much more difficult. You'll break a fibre cable here cause its so delicate, have a bag gbic there and then can't get the san wwn masks right - and you just had the most frustrating 3 weeks of your life...


    As you see, I would recommend worrying less about the performance and more about what you really need and what your environment looks like.
    If you just want performance for cheap, then local disks are unbeatable. Instead of spending money on expensive fibre or iscsi offload controllers buy tons of cheaper scsi cards, instead of running fibre and buying a fibre switch buy tons of disk drives.
    Most people make the mistake to worry about capacity - in reality, its the number of spindles you need to look at and the capacity or your storage is a result of that. Figure, each disk attached gives you about 100 i/o ops per second. If you need to do 5000 ops, you need 50 disks - no matter how they are connected. Figure a disk can get you 20 MB/sec - if you need 200MB/sec throughput, thats 10 disks.
    In that example, I need 50 disks then to satisfy my requirements. Next, take the max throughput and divide it by 3 - that's 66MB/sec for fibre then, 100MB for scsi and 33MB for iscsi over gigabit. So to run my 200MB/sec I'd need only 2 scsi channels, 3 fibre or 6 iscsi connections.
    Next of course can't have 50 drives on 2 channels - more than 3 disks drop the transfer rate dramatically... Since fibre and iscsi mask the physical spindles they don't care but I need to have 16 scsi controllers to really run the disks.

    Peter.
    • Peter, I did fail to include some important information. We're looking at an initial implementation in the 4TB range. The goals are: * Provide clustering ability * Snapshots for data protection * Easier/more efficient storage allocation * More reasonable backups On the clustering side, we're rolling out Exchange 2003 and Citrix in the next few months, and I'd like them to be highly available - hence clustering. We're also just simply wasting DAS space on each of about 35 total servers. Eventually, I s
      • We're also just simply wasting DAS space on each of about 35 total servers.


        Have you estimated a $$$ cost for this wasted disk space?

        Even if you factor whatever extra cost you want to associate managing DASD vs a SAN, do these extra costs justify the cost of a SAN?

        Ok iSCSI *might* be cheaper but does it improve the way you manage storage or does it just abstract the problem away from your host OS(es)?
      • [shameless plug]

        It sounds like your environment and requirements might be a good candidate for a Network Appliance filer... NetApp pioneered iSCSI, and you'd also get the benefit of instant snapshot backups, the fact that they already have (working) MPIO drivers for Windows, the performance of a RAID4 Fibre Channel storage system, as well as enterprise-level clustering/backup/recovery and a host of other bullet items you can find on your own.

        At risk of sounding like a dreaded sales person, expand your se
      • are lefthand and equallogic on the windows hardware compatibility lift? i know another follow up shamelessly plugs netapp, but i am fairly certain they are on the list. if the hw is not on the list, MSFT could tell you to pound sand if you have a support issue. (imagine you buy 3 million worth of HW, but for some reason exchange runs like a dog)

        ostiguy
  • The acid test: (Score:5, Informative)

    by hlygrail ( 700685 ) on Friday December 31, 2004 @01:31AM (#11225632)
    FC-AL is the "gold standard" for performance and reliability, but has limitations when you want to expand clustering (FC switches still cost gobs of $$$). Fibre Channel cabling also has distance limitations that go away with iSCSI.

    The acid test -- for me anyway -- is seeing LARGE customers (banks, airlines, government agencies [pick a 3-letter acronym], pharmaceuticals, major industries such as oil and energy, entertainment companies and movie studios, etc.) implementing iSCSI on an equally LARGE scale, and quite successfully.

    With few exceptions, if the underlying Ethernet network is functioning properly, iSCSI performs remarkably fast. No, you won't get the 2 Gbit FCAL rates -- *yet* -- but we have customers running dozens of large (>TB) databases off a single appliance over one or two GigE ports and iSCSI.

    Generally, it's recommended to segment off the iSCSI traffic so it's not routed or mixed with public traffic anyway, but even those (small) customers that pipe all of their storage appliance data through a single 10/100 Ethernet interface only have problems if they put too many users on there as well. (A direct crossover cable is *ideal* for iSCSI.)

    In addition, the Microsoft iSCSI initiator has finally outgrown its initial bugs/problems (with our help in some cases), and is darn solid with plenty of different targets.

    I'd love to drop a bunch of example company names, but I'm sure those companies consider that information to be competitive, and it's not my place to divulge it. Any large company you can think of already has an investment in FC-AL, and all but a very small percentage have iSCSI infrastructures as well. Medium-size and small (50 employees) companies are also seeing HUGE benefits from iSCSI their own implementations.

    The 0-day acid test (which works amazingly well in our labs with the right HBAs) is SAN booting over iSCSI. Imagine having an nnn-Terabyte set of storage, from which ALL your servers boot EVERYTHING. Not a single magnetic disk is required in the servers themselves. Makes server clustering and blade/grid computing so very attainable ... provided the network will drive it. 10Gbit Ethernet and more will definitely fuel this migration, sooner than you think!!

    (As an engineer for a major storage vendor (FC/iSCSI/near-line IDE storage/archiving), I work with all of this stuff on a daily basis. Not saying I'm an expert, just that I kinda know what I'm spouting here...)
    • Re:The acid test: (Score:2, Flamebait)

      by Nutria ( 679911 )
      Imagine having an nnn-Terabyte set of storage, from which ALL your servers boot EVERYTHING. Not a single magnetic disk is required in the servers themselves.

      That's called, "having all your eggs in 1 basket", and we all know what a bad idea that is...
      • Not necessarily.

        Having them all in one place makes mirroring the entire data set in real time more feasible. Heck, you can even mirror across a WAN link and have an offsite DR location that's always in sync with the original nnn TB data set.

        Having all the disks in one place can also allow for more security. Admins may have access to the servers, but not the media itself, so there's diminished risk in an entire hard disk going "missing," a la NASA JPL and/or Los Alamos. If you guard that, plus don't all
      • Let me guess, you're an MBA; you sir, should not comment on shit you know nothing about, even though that's likely your job. would you also say that having all your client-server communication done over the internet is having all your eggs in 1 basket? Yes, it's one set of storage, but the SAN has multiple fabrics, each logical disk having not a single point of failure. A sample path would be a protected raid device on the back, presented down two FAs, each connected to a separate fabric, each fabric con
        • you're an MBA

          Don't pull shit out your ass. I'm a DBA, with a BSci in Comp. Sci.

          We have a Hitatchi SAN, and one day, guess what? The SAN crashed!! All systems using it were down for a day: AS/400s, HP PA-RISC boxen, large Alpha VMS servers, and mainframes.

          So fuck you and the horse you rode in on.
      • Re:The acid test: (Score:3, Insightful)

        by hbackert ( 45117 )

        That's called, "having all your eggs in 1 basket", and we all know what a bad idea that is...

        About the booting part: at work we boot from local disks because we Unix SAs don't have control of the EMCs, thus if a machine does not boot, we cannot do much beside calling someone who has no idea about Sparc machines. If we Unix SAs were able to control the EMCs and everything related like the FC switches, then we would boot directly from SAN. If I were able to control the iSCSI storage box, the routers and s

      • The "all your eggs in one basket" metaphor only works if all the baskets are equal. And if you are comparing iSCSI/FC to internal storage, you're talking about completely different 'baskets'.

        I could have a lot of fun with that metaphor but i'm sure you get the point without it.
  • by mkcmkc ( 197982 ) on Friday December 31, 2004 @01:54AM (#11225721)
    An alternative benchmark to consider: With old-fashioned direct attached SCSI, you can pretty much only screw up the disks one host at a time. With FC (and iSCSI I suppose), you can completely fuck up your hundred-million-dollar site with one mouse click. Now that's power!

    Mike

    • Almost...

      iSCSI LUNs can be pretty friggin' huge. More than once, I've seen a customer nuke the holy bejeebers out of a 950GB iSCSI LUN all at once, with a single click (the equivalent of "LUN DESTROY"). I actually had a WebEx session to a customer once and watched him delete a 1.3TB LUN before I could stop him.

      If you're on plain JBOD, you have to get out your restore tapes and pray to the tape gods. If you're on a SAN/NAS appliance (like NetApp), you can usually restore that data in a matter of seconds
  • by grantsucceeded ( 126212 ) on Friday December 31, 2004 @02:35AM (#11225874)
    The original question was regarding real world performance of iSCSI in particular, and since frew of the posts seem to touch on that I may as well tell what I've learned from hard experience regarding the other technologies: SCSI and FCAL.

    My experience is with very high transaction volume OLTP databases (oracle) backing a financial website. I've found that neither SCSI nor FCAL adapters limit performance significantly. This was with qlogic qla3200 adapters, or with highend adaptec Ultra320, on Solaris 9 and the last few versions if RedHat enterprise. Only the older versions of redhat had some kind of problem with the qlogic driver, plus bounce buffer IO, which drove down performance. But then to be nit picky, that was the driver not the HBA. Solaris was always fine, and now redhat is too.

    The main performance challenge was *always* tuning the database and spreading out on lots of spindles. The HBAs at over 200M/sec each never posed a problem on larger sun boxes (8 or more procs) with 7 or 8 way parallel sequential reads going. On smaller hosts or smaller disk arrays,k the problem was always on the host itself or the disk seek times respectively, not the hbas themselves.

    A 10k rpm drive will do about 70 mbytes/sec off the outer platter (near block 0) and as a rule of thumb, a 2Gbit fcal adapter will do 200mbytes a second (at least on solaris or newer redhat EL). So my dual qlogics would do 400Mbytes a second under absolute optimal disk access, but typically it's not that perfect 8 way parallel *serial* scan off the outer platter, its usually farily random.

    So in the high end database applications (datawareyouse or OLTP) least the usual tuning challenge (and $$ for that matter) are with getting a fat spread across a lot of spindles, and making sure the application is either caching well (OLTP) or doing orderly, sequential scans (datawarehouse)
  • "Hi. I'm thinking about buying a car. Which is the best one?"

    Ummm What are you doing with it? What's your budget, and what are your expectations? Some people think a TB is big, some consider 20+ TB a building block standard (I do).

    1. Look on the net and in mags. - read read read
    Is this a big implementation, or a small one. I'd bet a small one, or you probably wouldn't be asking here - that probably means you don't want iscsi. Direct FCAL on multi-port RAIDs might be your speed - if you aren't big en
  • Has anyone seen a reverse-iSCSI adapter? Sorry, this isn't the best possible name, I'm sure.

    I've seen plenty of adapters that you can hook to a SCSI device and call it an iSCSI device but that assumes you have a proper iSCSI HBA.

    What I need is something I can attach to a SCSI HBA that has an ethernet connector on the other side, so I can connect it to a network and do iSCSI over it.

    This particular instance is for an old machine that noone's ever going to make an iSCSI HBA for to connect it to a tape dri

"If it ain't broke, don't fix it." - Bert Lantz

Working...