Does ZFS Obsolete Expensive NAS/SANs? 578
hoggoth writes "As a common everyman who needs big, fast, reliable storage without a big budget, I have been following a number of emerging technologies and I think they have finally become usable in combination. Specifically, it appears to me that I can put together the little brother of a $50,000 NAS/SAN solution for under $3,000. Storage experts: please tell me why this is or isn't feasible." Read on for the details of this cheap storage solution.
Get a CoolerMaster Stacker enclosure like this one (just the hardware not the software) that can hold up to 12 SATA drives. Install OpenSolaris and create ZFS pools with RAID-Z for redundancy. Export some pools with Samba for use as a NAS. Export some pools with iSCSI for use as a SAN. Run it over Gigabit Ethernet. Fast, secure, reliable, easy to administer, and cheap. Usable from Windows, Mac, and Linux. As a bonus ZFS let's me create daily or hourly snapshots at almost no cost in disk space or time.
Total cost: 1.4 Terabytes: $2,000. 7.7 Terabytes: $4,200 (Just the cost of the enclosure and the drives). That's an order of magnitude less expensive than other solutions.
Add redundant power supplies, NIC cards, SATA cards, etc as your needs require.
Specifics please. (Score:5, Insightful)
Unless you would like to give more specifics. Cause I am going to say in 99% of cases where you want fast, reliable, and cheap storage you only get to pick two.
Re:Everyman? (Score:4, Insightful)
I don't have a proper video camera, but I'm sure that people who do, have even bigger storage requirements.
Not only that, what with all the music you can copy of a friends HD now, your storage just jumps a bit more! (I've got literally more then 10 gigabytes of music on my desktop HD. And I know people who have hundreds of CDs, so if they ripped all those, they would have much more...)
Added to all those movies you can either rip or download...
Chuck in a decent network, family and/or friends, and you can now stream all this stuff around to wherever you want it.
I'd say then, that the most common use of all this space, multimedia. Not sure who has terrabytes of multimedia though.
Re:Everyman? (Score:4, Insightful)
80GB DVDs
120GB games
14GB/hr for DV editing
1 whole drive for OSes
RAID-5ed (1 parity drive)
So I'm up to 4 200gb drives right now, without even trying hard.
Soon I'm going to jump to 500GB drives, and I expect to be hitting their limits in a year or so.
Also, how the hell am I supposed to back up all this?! Incrementals would be 10gb+ / week
Re:Its just not the same thing. (Score:3, Insightful)
I mean, you could deploy 10 times as many of these as you could your array, giving me 10 times the storage.
Depending on what you do, it has to hit the network sometime. For example - we have a big expensive SAN solution. What's it used for? As a massive shared drive.
For a fraction the price we could of put a similarly sized one of these in every organization. Sure, it wouldn't be able to serve quite as much data, but most of our stuff isn't accessed extremely often anyways.
I've noted before that in many cases it'd be cheaper for us to backup to IDE hard drives than to tape. The tape alone costs more per megabyte than a HD, and has slower transfer rates to boot. Our backup solution could be several of these, connected by fiber to another building(so we don't have to move them).
Re:Everyman? (Score:3, Insightful)
I've got just over a terabyte of live storage around the house and I probably use about half of it - I have a couple of hundred gigs of video and about 60 gigs of music. I know of someone who is currently buying seven of Hitachi's new terabyte HDs for an in-home video streaming system, There's always someone who has a use for it.
Re:ok for low end, not for high (Score:5, Insightful)
I've seen plenty iSCSI of solutions coupled with NAS servers that get pretty good throughput in this price range that are already integrated and ready to go, but the bottom line is that if you want high-peformance, high-availability storage for I/O-intensive applications, you need a fiber SAN/NAS solution.
Re:Specifics please. (Score:4, Insightful)
Do you honestly believe the slogan of "business-grade"? Come on, let the marketing jargon go. Hardware designs are expensive, so rarely are there multiple designs. Sales guys are selling you additional support, but the hardware is rarely different. If it is, then the volume is not there, so the reliability is actually worse. Volume is the king of reliability. Reliability is always more dependent on the age of the design and its volume rather than the intended customer...
Re:ok for low end, not for high (Score:5, Insightful)
I agree. At my work, we have a SAN ... low-end frames (SATA) to mid-range (FC+SATA) to high-end frames (FC.) We put a front-end on the low-end and mid-range storage using a NAS, so you can still access using the storage fabric or over IP delivery. Having a SAN was a good idea for us, as it allowed us to centralize our storage provisioning.
I'm familiar with ZFS and the many cool features laid out in this Ask Slashdot. The simple answer is: ZFS isn't a good fit to replace expensive SAN/NASs. However, ZFS on a good server with good storage might be a way to replace an inexpensive SAN/NAS. Depending on your definition of "inexpensive." And if you don't mind the server being your single point of failure.
Re:Specifics please. (Score:5, Insightful)
I disagree completely. Computer hardware is a commodity. The big box makers are afraid of this very kind of configuration which would blow them out of business if more people caught on to it. No, they use FUD to convince PHBs that because of the low cost, it cannot possibly be as good. Hot-swap and hot-spare are commodity technologies. But, please, feel free to continue the FUD, because it helps the bottom line.
Re:Its just not the same thing. (Score:3, Insightful)
also show me a SINGLE sata or ide drive that can touch the data io rates of a u320 scsi drive with 15K spindle speeds.
Low end consumer drive cant do the high end stuff. Dont even try to convince anyone of this. guess what, those uses are not anywher near strange for big companies. witha giant SQL db you want... no you NEED the fastest drives you can get your hands on and that is SCSI or Fiberchannel.
No (Score:3, Insightful)
Potentially it will obselete low-end NAS/SAN hardware (eg: Dell/EMC AX150i, StoreVault S500) in the next couple of years, for companies who are prepared to expend the additional people time in rolling their own and managing it (a not insignificant cost - easily making up $thousands or more a year). There's a lot of value in being able to take an array out of a box, plug it in, go to a web interface and click a few buttons, then forget it exists.
However, your DIY project isn't going to come close to the performance, reliability and scalability of even an off the shelf mid-range SAN/NAS device using FC drives, multiple redundant controllers and power supplies - even if the front end is iSCSI.
Not to mention the manageability and support aspects. When you're in a position to drop $50k on a storage solution, you're in a position to be losing major money when something breaks, which is where that 24x7x2hr support contract comes into play, and hunting around on forums or running down to the corner store for some hardware components just isn't an option.
ZFS also still has some reliability aspects to work out - eg: hot spares. Plus there isn't a non-OpenSolaris release that offers iSCSI target support yet AFAIK.
I've looked into this sort of thing myself, for both home and work - and while it's quite sufficient for my needs at home, IMHO it needs 1 - 2 years to mature before it's going to be a serious alternative in the low-end NAS space.
Re:Reliable? (Score:3, Insightful)
Re:Its just not the same thing. (Score:2, Insightful)
SCSI vs IDE is no issue. Newer technology drives are better than older drives is a bigger factor. So SATA lasts as long as SCSI. Performance wise for sequential access, a 500GB 7200RPM disk beats a 10000 RPM 72GB or 144 GB SCSI any day. For my laptop, my 160GB 5400 RPM disk is faster than the 7200 RPM 100GB disk.
The advantage of the SCSI disk is native command queuing. So that it can stop on the way from sector A to C and read sector B. But, this is also implemented in SATA-300 drives, so this advantage is also gone from SCSI.
I have been a big SCSI fanboy myself, but the magic has gone. SATA has flown past SCSI.
Re:Its just not the same thing. (Score:3, Insightful)
However, the price point on 15k drives is such a comparison for a single drive vs multiple drives is reasonable. The basis is $$/GB, not drive A vs drive B.
Ask Google how they get their throughput on their terrabyte datasets. Hint: it's not due to 15k drives.
Re:Reliable? (Score:3, Insightful)
At $3000 for everything it would be logical to stock a spare motherboard, Power supply, NIC, controller cards, and A few Hard drives plus a few hot spares.
With SMART it isn't hard and other sensor packages it wouldn't be hard to build in monitoring.
It could be done but it would be a project. So yes it could be done but for a business buying a SAN will be more logical if for no other reason than time. It would take time to set up a system. It could be cheaper and have just as little downtime as a commercial SAN but it could also be a huge time sink. For a small tech company it might be a great solution and a potental product. For a HUGE tech company it might be a good solution and a potental product. For anybody else it is a risk probably not worth taking.
Re:Specifics please. (Score:4, Insightful)
Re:Specifics please. (Score:5, Insightful)
1) That case has twelve spindles. You aren't going to get the same performance from a dozen drives as you get from an hundred.
2) That system includes a small Celeron D processor with 512 MB RAM. You aren't going to get the same performance as you'll get from multiple dedicated RAID controllers with twenty+ gig of dedicated disk cache.
3) Your single gigabit ethernet interface won't even come close to the performance of the three or four (or ten) 2 gigabit fibre channel adapters involved in most SAN arrays.
4) Software iscsi initiators and targets aren't a replacement for dedicated adapters.
5) The Hitachi array at work never goes down. Ever. Not for expansions. Not for drive replacements. Not for relayouts. The reliability of a PC and the opportunity to do online maintenance won't approach that of a real array.
Don't get me wrong. That case makes me all tingly inside -- for personal use. But as a SAN replacement, fuck no. It's not the same thing. The original question just shows ignorance of what SANs are and the roles they fill in the data center.
As a workgroup storage solution for a handful of end users on their desktops, that solution probably may be a good fit. As a storage solution for ten (or two hundred) business critical server systems, no way.
Re:Hardware RAID (Score:1, Insightful)
ZFS isn't SAN capable (Score:2, Insightful)
From the ZFS FAQ:
Q: Suppose I have three E450s. Would ZFS allow me to integrate storage across all three boxes into one big "poor man's SAN"?
A: No, ZFS is a local filesystem (for the time being). To access storage attached to a different host, use NFS.
Re:Real SANs do more (Score:3, Insightful)
I'm not sure where you're shopping, but I've only seen a handful of 10Gb ethernet switches, and they have all been dreadfully expensive. So factor in a pair of those, plus multiple 10Gb TOE cards, and I bet that your price isn't any cheaper than fiber. In fact, it's probably more expensive.
It makes huge sense to use the same technology for your LAN & SAN.
Not always, and I think that this tends to be overstated by the iSCSI proponents. This statement makes perfect sense on the surface, but when you look into the details it comes up short. It makes sense from a knowledge transfer standpoint to use the same technology. You can get by with a single person who understands TCP/IP networking instead of a person who does networking and a person who does SAN networking. But that person who handles the networking for the SAN usually also handles all of the zoning, space allocation, LUN carving, and everything else that comes with managing a SAN. Who does that now?
Then there's the whole issue of network segregation. Do you really want to put your SAN on the same switches are your desktop PCs or servers? Probably not. It doesn't make a lot of sense. Sure, you can VLAN it off, but those switches still have a finite amount of bandwidth available, and you don't want I/O intensive applications (like iSCSI SANs) eating up all of the bandwidth that users are accustomed to having. Then there's the speed issue. Do you buy a ton of 10Gb switches for all of your datacenter network, or just a few for the iSCSI SAN? Do you run iSCSI on 1Gb switches instead? What you're going to end up doing eventually is just buying dedicated networking hardware for the SAN and physically separating it from the rest of your network. So at that point, what advantages do you get with iSCSI over fiber?
Re:ZFS (Score:1, Insightful)
Re:Specifics please. (Score:4, Insightful)
Solly Chollie.
We have one of those "$50,000 SANs". Actually, with the expansion, ours cost closer to $110,000, but whatever. For one, it's got more drives (I think we're up to 30). For two, they're 10k RPM SCSI drives. For three, it's fibrechannel, and all the servers that it's connected to get full speed access to the drives. For four, we have a service level agreement with Dell/EMC (It's an EMC array, rebranded under Dell's name) that says if we have a dead drive, we get a dell service hobbit on site within 4 hours with a new drive. We also get high level support from them for things like firmware upgrades (we had to go through that recently to install the expansion).
The grandparent is exactly correct:
fast, reliable, and cheap storage - you only get to pick two.
We picked fast and reliable. You picked reliable and cheap. There's nothing wrong with either decision, but please don't tell me that a $3000 solution would have done everything that we needed.
You can't have all three.
~Wx
Re:ZFS (Score:5, Insightful)
Because they are just filesystems. ZFS is also a volume manager.
> Or a CentOS based OS in place of Solaris?
Because CentOS doesn't have ZFS.
+4 Interesting. Awesome.
Quality - industrial vs. commercial (Score:5, Insightful)
I worked at ATMEL many years ago in their EPROM division. I had an up close and personal view of the screening flows, both Military and otherwise. Let's put aside the issue of Military screening, which is extensive and costly. You can't make very much out of Military grade ICs, because there are not very many available.
The difference between commercial and industrial parts is one of operating temperature, not quality. (In point of fact, there was no actual difference in the screening or handling.) The quality standards for both parts were the same - the goal was always zero defects. I spent weeks weeding out a problem with a 50 ppm failure rate that was slipping through our screening, and everyone was damned happy when I fixed it.
There's no reason to expect a correlation between maximum operating temperature and quality. A part might run too slow at elevated temperature to pass, but this will usually happen for process variation reasons that do not affect the expect lifetime of the part.
Any part coming from a reputable IC manufacturer should have the same level of quality, regardless of the rating.
Now, that being said, there is a very serious quality issue that an OEM does need to address, and that's counterfeit parts. If an OEM is not careful about where their parts come from, or buys them cheap and looks the other way, then there quality will obviously suffer. But this isn't so much a commercial versus industrial quality; it's about honest versus dishonest business practices.
Re:ZFS (Score:5, Insightful)
Back to the question. Can it replace a SAN. Depends on the required performance. If you have 25 or 30 video edit workstations or a corporation with 5,000 desk tops it's hard to see how one Solaris server is going to work. You need something that can a lot of IO bandwidth.
Re:ZFS (Score:1, Insightful)
ZFS loves cheap disks.
And with only two commands (zpool and zfs) and integrated web frontend, it's very simple to use.
Re:Have you tried out Starfish? (Score:4, Insightful)
We do that as a preventative measure for people that don't get support through us. We don't want anybody to assume that the file system is ready for a highly-available cluster without talking to us first. As with all file systems, there are trade-offs to using Starfish. We are being honest - the software is stable as far as we can tell, but it doesn't have a great deal of field use (it was released to the public in March 2006).
We must be especially careful with file systems - data is very important to people. If somebody uses our system and loses data, we can't fix that - not that it has ever happened. We put the beta message as a warning that people should talk to us before thinking about putting our system into production. After all - it doesn't have nearly the amount of testing behind it that EXT3 does. It is common for a file system to remain as a beta product for the first year or two.
Hmm... ZFS and Starfish aren't really meant to address the same storage problem. Take a bit of time and read through what ZFS does and what Starfish does. ZFS is a block-level file system. Starfish is a file-level distributed clustered storage system. Those are two very different things - at the end of the day they store files, but in very different ways and for very different purposes. Starfish can use ZFS as it's block-level file system... it can also use Reiser and EXT3.
Here are a couple of reasons to use Starfish (even though we think that ZFS is a fantastic solution for block-level file system problems):
However, this really isn't a "what is better, Starfish or ZFS?" discussion. You can have the best of both worlds: Starfish as the file-level network storage cloud using ZFS as the block-level file system.
-- manuRe:ZFS (Score:2, Insightful)
Re:The best laid plans of mice and men... (Score:2, Insightful)
Re:ZFS (Score:3, Insightful)
Re:ZFS (Score:3, Insightful)