Slashdot Log In
Does ZFS Obsolete Expensive NAS/SANs?
Posted by
kdawson
on Wed May 30, 2007 07:02 AM
from the fast-secure-reliable-cheap dept.
from the fast-secure-reliable-cheap dept.
hoggoth writes "As a common everyman who needs big, fast, reliable storage without a big budget, I have been following a number of emerging technologies and I think they have finally become usable in combination. Specifically, it appears to me that I can put together the little brother of a $50,000 NAS/SAN solution for under $3,000. Storage experts: please tell me why this is or isn't feasible." Read on for the details of this cheap storage solution.
Get a CoolerMaster Stacker enclosure like this one (just the hardware not the software) that can hold up to 12 SATA drives. Install OpenSolaris and create ZFS pools with RAID-Z for redundancy. Export some pools with Samba for use as a NAS. Export some pools with iSCSI for use as a SAN. Run it over Gigabit Ethernet. Fast, secure, reliable, easy to administer, and cheap. Usable from Windows, Mac, and Linux. As a bonus ZFS let's me create daily or hourly snapshots at almost no cost in disk space or time.
Total cost: 1.4 Terabytes: $2,000. 7.7 Terabytes: $4,200 (Just the cost of the enclosure and the drives). That's an order of magnitude less expensive than other solutions.
Add redundant power supplies, NIC cards, SATA cards, etc as your needs require.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
ZFS (Score:5, Informative)
Re:ZFS (Score:4, Interesting)
Parent
Re:ZFS (Score:5, Insightful)
Because they are just filesystems. ZFS is also a volume manager.
> Or a CentOS based OS in place of Solaris?
Because CentOS doesn't have ZFS.
+4 Interesting. Awesome.
Parent
Re:ZFS (Score:5, Insightful)
Back to the question. Can it replace a SAN. Depends on the required performance. If you have 25 or 30 video edit workstations or a corporation with 5,000 desk tops it's hard to see how one Solaris server is going to work. You need something that can a lot of IO bandwidth.
Parent
Re:ZFS (Score:5, Informative)
A correction:
Copy-on-write is quite a misnomer here (even if Sun use that term). It is a Transactional filesystem. Blocks are not copied upon write, they are only written and then the transaction log is updated. It's far more clever than old-fashined COW schemes. It can be compared with NetApp's WAFL filesystem.
Parent
Re:ZFS (Score:5, Interesting)
Mass of the earth = 5.9742 × 10^27 grams
Make the drives out of the earth, you need a drive density of 57Gb/gram
A drive with a density of 1 bit per carbon atom, 5.4 *10^10 metric tons
Size of said nanotech drive, a cube 2.88 Km tall (at the standard density of carbon)
Never in your lifetime is a really safe bet.
T
Parent
Re:ZFS (Score:4, Informative)
Increased reliability (all data is checksummed, even in non-raid configurations), near brainless management (e.g., newfs is not needed, raid configurations are trivial to setup, etc.), built-in optional compression (even for swap, if you're feeling masochistic), etc.. Encryption is in development.
See my other posts here for links.
Parent
Specifics please. (Score:5, Insightful)
Unless you would like to give more specifics. Cause I am going to say in 99% of cases where you want fast, reliable, and cheap storage you only get to pick two.
Re:Specifics please. (Score:5, Insightful)
I disagree completely. Computer hardware is a commodity. The big box makers are afraid of this very kind of configuration which would blow them out of business if more people caught on to it. No, they use FUD to convince PHBs that because of the low cost, it cannot possibly be as good. Hot-swap and hot-spare are commodity technologies. But, please, feel free to continue the FUD, because it helps the bottom line.
Parent
Re:Specifics please. (Score:4, Insightful)
Parent
Re:Specifics please. (Score:5, Informative)
No system can compensate for bad management by people, but I digress.
All data is critical. But, to say that your data is less safe with a system that cost $4700 than a system that cost $50,000 is fallacious without some heavy proof behind it. For now, I am going to ignore that a functional backup is part of "mission critical" and just address the online storage portion of the argument.
Let's start with a server white box. Something with redundant power supplies, ethernet, etc. Put a mirrored boot drive in it. Install Linux. So far, the cost is fairly low. Add an external disk array, at least 15 slots, the ones with hot-swap, hot-spare, RAID 5, redundant power supplies and fill it with inexpensive (but large) SATA drives. Promise sells one, as do others. Attach to server, voila, a cheaper solution than EMC for serving up large amounts of disk space.
What if a drive fails? The system recreates the data (it is RAID5, after all) onto a hot-spare. You remove the bad drive, insert new, run the administration. The uses won't even notice their MP3's and Elf Bowling game were ever in danger.
For the people who believe strongly in really expensive storage solutions, please explain why. I would like to know if you also hold the same theory for your desktop PCs, because surely, a more expensive PC has to be better. Right?
Parent
Re:Specifics please. (Score:4, Informative)
It's the same reason we buy Dell. We could buy white boxes or parts from Newegg for all of our systems, but talk about a hassle when it comes to them needing hardware maintenance or just assembly. With the support Dell offers, we get a complete box that's been tested, we just need to reformat and install our own stuff. Something breaks, we make a 10 minute phone call and get a replacement the next day, with or without assisted installation. But we pay probably 30% more per box for that.
Parent
RAID controller failure (Score:5, Informative)
Because I've seen that happen more than once.
I'm not saying the more expensive solution is better. I'm just saying that in my personal experience I've seen *more* data destroyed from RAID controller failure than from hard drive failure. I would love to find out the solution to that one.
I do not claim to be a hardware expert or system administrator, so there may be a well known solution (don't buy 'brand X' RAID controllers). I just don't happen to know it.
Parent
Re:Specifics please. (Score:5, Interesting)
The premium paid for higher-end storage is decidedly nonlinear. For marginally more reliable or faster storage, you pay about a factor of ten. One example I'm familiar with is Hitachi. We had a 64TB HDS array a few years ago that was worth roughly $2M. We could have purchased an equivalent amount of commodity storage for probably $200k at the time, but didn't. Why would we spend the extra money? Speed, configurability, expandability, and reliability.
First of all, speed. That thing was loaded with 73GB 15k FCAL drives. RAID was in sets of four disks, with no two disks in a set sharing the same controller, backplane, or cache segment. Speaking of cache, the rule was 1GB/TB. so we had 64GB of fast, low-latency, fully mirrored cache on the thing. It was insanely fast, and (most importantly) didn't slow down under point load. One tool automatically ran on the array itself, looking for hotspots and reallocating data on the fly.
Configurability: We could mirror data synchronously or asynchronously to our DR site, by filesystem, file, block, LUN, or byte. We could dynamically (re)allocate storage to multiple systems, and moving databases between machines was a breeze. Disk could be allocated from different pools (i.e. different performing drives could be installed), depending on requirements. Quality-of-Service restrictions could be put in place as well, although we never used them.
Expandability: The beast had 32 pairs of FC connections, could support 96GB of internal mirrored cache, and I can't remember how much actual disk. The key wasn't the amount of disk we could put on it, so much as how well the bandwidth scaled--and it scaled well.
Finally, the real key - Reliability. All connections were dual-pathed, with storage presented to a pair of smart FC switches which were zoned to present storage to various systems. We could lose three of the four power cables to the main unit (auxiliary disk cabinets only had two power connections each), and still run. We could lose any entire rack, and still run. We could lose any switch in our environment, and still run. We could lose two disks from the same RAID set and still run. When we lost a disk, the system would automatically suck up some cache to use for remirroring the data to multiple disks as fast as possible, and then after protecting it, would remirror back to a single logical device. In the event that we lost the entire device, we could run from our DR site synchronous mirror with less than a ten second failover.
This sort of thing is massive overkill for most people and companies, but when someone is doing realtime commodities trading, (or banking, or stock exchanges, etc.) the protection and support are worth the extra money. You just can't build that sort of thing on your own for any less money, at the end of the day.
Parent
Re:Specifics please. (Score:4, Insightful)
Solly Chollie.
We have one of those "$50,000 SANs". Actually, with the expansion, ours cost closer to $110,000, but whatever. For one, it's got more drives (I think we're up to 30). For two, they're 10k RPM SCSI drives. For three, it's fibrechannel, and all the servers that it's connected to get full speed access to the drives. For four, we have a service level agreement with Dell/EMC (It's an EMC array, rebranded under Dell's name) that says if we have a dead drive, we get a dell service hobbit on site within 4 hours with a new drive. We also get high level support from them for things like firmware upgrades (we had to go through that recently to install the expansion).
The grandparent is exactly correct:
fast, reliable, and cheap storage - you only get to pick two.
We picked fast and reliable. You picked reliable and cheap. There's nothing wrong with either decision, but please don't tell me that a $3000 solution would have done everything that we needed.
You can't have all three.
~Wx
Parent
Re:Specifics please. (Score:5, Insightful)
1) That case has twelve spindles. You aren't going to get the same performance from a dozen drives as you get from an hundred.
2) That system includes a small Celeron D processor with 512 MB RAM. You aren't going to get the same performance as you'll get from multiple dedicated RAID controllers with twenty+ gig of dedicated disk cache.
3) Your single gigabit ethernet interface won't even come close to the performance of the three or four (or ten) 2 gigabit fibre channel adapters involved in most SAN arrays.
4) Software iscsi initiators and targets aren't a replacement for dedicated adapters.
5) The Hitachi array at work never goes down. Ever. Not for expansions. Not for drive replacements. Not for relayouts. The reliability of a PC and the opportunity to do online maintenance won't approach that of a real array.
Don't get me wrong. That case makes me all tingly inside -- for personal use. But as a SAN replacement, fuck no. It's not the same thing. The original question just shows ignorance of what SANs are and the roles they fill in the data center.
As a workgroup storage solution for a handful of end users on their desktops, that solution probably may be a good fit. As a storage solution for ten (or two hundred) business critical server systems, no way.
Parent
Re:Specifics please. (Score:4, Insightful)
Do you honestly believe the slogan of "business-grade"? Come on, let the marketing jargon go. Hardware designs are expensive, so rarely are there multiple designs. Sales guys are selling you additional support, but the hardware is rarely different. If it is, then the volume is not there, so the reliability is actually worse. Volume is the king of reliability. Reliability is always more dependent on the age of the design and its volume rather than the intended customer...
Parent
Re:Specifics please. (Score:4, Informative)
Parent
Re:Specifics please. (Score:4, Informative)
True, but there is a difference. The difference is in QA.
The "consumer-grade" and "business-grade" are the same off the shelf stuff, but if you are getting business-grade stuff from a reputable vendor they QA the consumer-grade parts, throw out the bad ones, and stamp "business-grade" on the ones that survive. This is why the business-grade level of products often are a generation or so behind the consumer-grade level.
Yes, you can get lucky and get consumer-grade stuff that works great. But if it doesn't, then you are the QA guy, and the downtime is on you. If the time for you to do the QA and the associated downtime is cheaper than the cost of business-grade, then by all means do it. Otherwise, you have to pay the extra bucks.
Now, regarding NASs, I think these things are overpriced, especially the maintenance on them. The maintenance goes through the roof once the equipment is beyond the MTBF of the drives, which is where a high dollar NAS should shine right? Any piece of crap RAID box will work when all the drives are new and functioning well. What you are paying for is the redundancy and availability, which is redundant to pay for when all of the equipment is new.
Parent
Quality - industrial vs. commercial (Score:5, Insightful)
I worked at ATMEL many years ago in their EPROM division. I had an up close and personal view of the screening flows, both Military and otherwise. Let's put aside the issue of Military screening, which is extensive and costly. You can't make very much out of Military grade ICs, because there are not very many available.
The difference between commercial and industrial parts is one of operating temperature, not quality. (In point of fact, there was no actual difference in the screening or handling.) The quality standards for both parts were the same - the goal was always zero defects. I spent weeks weeding out a problem with a 50 ppm failure rate that was slipping through our screening, and everyone was damned happy when I fixed it.
There's no reason to expect a correlation between maximum operating temperature and quality. A part might run too slow at elevated temperature to pass, but this will usually happen for process variation reasons that do not affect the expect lifetime of the part.
Any part coming from a reputable IC manufacturer should have the same level of quality, regardless of the rating.
Now, that being said, there is a very serious quality issue that an OEM does need to address, and that's counterfeit parts. If an OEM is not careful about where their parts come from, or buys them cheap and looks the other way, then there quality will obviously suffer. But this isn't so much a commercial versus industrial quality; it's about honest versus dishonest business practices.
Parent
Let's pretend you're right for a moment... (Score:5, Interesting)
So, that's $6k total instead of $3k.
The one problem I have with OCFS2 is that when it fences a system, it tends to either bring the whole thing down (kernel panic), or in newer versions, give you the option of forceably rebooting instead. This killed it for a project I was working on, where one of the machines had other mission-critical systems running that were not on the OCFS2, and thus, it seemed retarded to panic and bring down everything else too.
So if that's your problem, you can always build a third, identical system to run the other stuff on. $9k.
Even if you figure another $1k for random stuff, like maybe a LOT of gigabit crossovers, or 10gig fiber, or something, that's still a fifth of the cost of the "business-grade" or whatever else he was considering. Even assuming the worst-case scenario, where the homebrew system costs a lot more to maintain (even electricity and cooling, maybe), how long will it take for it to cost another $40k? And this way, you have an ENTIRELY redundant system -- the only way you lose it is if, say, the whole building blows up.
I mean, I sort of agree that you get what you pay for. But when the difference in price is that much, the only way it's ever worth it is if there's really great support with the high-end package. And is it $40k worth of support? If not, I imagine this guy could put together a company selling little $3k, $6k, and $10k systems for $20k each (including support), shaving off $30k even for the most paranoid.
And all of that is pretending you're right about the cheap consumer-grade hardware actually being less reliable.
Parent
Re:I'm a Hardware Guy, Not a ZFS Guy (Score:5, Informative)
OK:
- ZFS w/flaky hardware (scary): http://blogs.sun.com/elowe/entry/zfs_saves_the_da
y _ta [sun.com]
- Self Healing with ZFS: http://www.opensolaris.org/os/community/zfs/demos
/ selfheal/ [opensolaris.org]
- 100 Mirrored Filesystems in 5 minutes: http://www.opensolaris.org/os/community/zfs/demos
/ basics/ [opensolaris.org]
And, for more than you wanted to know about ZFS: http://en.wikipedia.org/wiki/ZFS [wikipedia.org]Parent
Congradulations, you discovered the "File Server" (Score:5, Informative)
BBH
ok for low end, not for high (Score:4, Informative)
Re:ok for low end, not for high (Score:5, Insightful)
I've seen plenty iSCSI of solutions coupled with NAS servers that get pretty good throughput in this price range that are already integrated and ready to go, but the bottom line is that if you want high-peformance, high-availability storage for I/O-intensive applications, you need a fiber SAN/NAS solution.
Parent
Re:ok for low end, not for high (Score:5, Insightful)
I agree. At my work, we have a SAN ... low-end frames (SATA) to mid-range (FC+SATA) to high-end frames (FC.) We put a front-end on the low-end and mid-range storage using a NAS, so you can still access using the storage fabric or over IP delivery. Having a SAN was a good idea for us, as it allowed us to centralize our storage provisioning.
I'm familiar with ZFS and the many cool features laid out in this Ask Slashdot. The simple answer is: ZFS isn't a good fit to replace expensive SAN/NASs. However, ZFS on a good server with good storage might be a way to replace an inexpensive SAN/NAS. Depending on your definition of "inexpensive." And if you don't mind the server being your single point of failure.
Parent
And more importantly. (Score:5, Funny)
Current issues (Score:5, Informative)
HW Raid vs ZFS (Score:5, Informative)
Parent
vs Reiser4 (someday, maybe) (Score:5, Informative)
Reiser4 had the same problems with fsync -- basically, fsync called sync. This was because their sync is actually a damned good idea -- wait till you have to (memory pressure, sync call, whatever), then shove the entire tree that you're about to write as far left as it can go before writing. This meant awesome small-file performance -- as long as you have enough RAM, it's like working off a ramdisk, and when you flush, it packs them just about as tightly as you can with a filesystem. It also meant far less fragmentation -- allocate-on-flush, like XFS, but given a gig or two of RAM, a flush wasn't often.
The downside: Packing files that tightly is going to fragment more in the long run. This is why it's common practice for defragmenters to insert "air holes". Also, the complexity of the sync process is probably why fsync sucked so much. (I wouldn't mind so much if it was smarter -- maybe sync a single file, but add any small files to make sure you fill up a block -- but syncing EVERYTHING was a mistake, or just plain lazy.) Worse, it causes reliability problems -- unless you sync (or fsync), you have no idea if your data will be written now, or two hours from now, or never (given enough RAM).
(ZFS probably isn't as bad, given it's probably much easier to slice your storage up into smaller filesystems, one per task. But it's a valid gotcha -- without knowing that, I'd have just thrown most things into the same huge filesystem.)
There's another problem with reliability: Basically, every fast journalling filesystem nowadays is going to do out-of-order write operations. Entirely too many hacks depend on ordered writes (ext3 default, I think) for reliability, because they use a simple scheme for file updating: Write to a new temporary file, then rename it on top of the old file. The problem is, with out-of-order writes, it could do the rename before writing the data, giving you a corrupt temporary file in place of the "real" one, and no way to go back, even if the rename is atomic. The only way to get around this with traditional UNIX semantics is to stick to ordered writes, or do an fsync before each rename, killing performance.
I think the POSIX filesystem API is too simplistic and low-level to do this properly. On ordered filesystems, tempfile-then-rename does the Right Thing -- either everything gets written to disk properly, or not enough to hurt anything. Renames are generally atomic on journalled filesystems, so either you have the new file there after a crash, or you simply delete the tempfile. And there's no need to sync, especially if you're doing hundreds or thousands of these at once, as part of some larger operation. Often, it's not like this is crucial data that you need to be flushed out to disk RIGHT NOW, you just need to make sure that when it does get flushed, it's in the right order. You can do a sync call after the last of them is done.
Problem is, there are tons of other write operations for which it makes a lot of sense to reorder things. In fact, some disks do that on a hardware level, intentionally -- nvidia calls it "native command queuing". Using "ordered mode" is just another hack, and its drawback is slowing down absolutely every operation just so the critical ones will work. But so many are critical, when you think about it -- doesn't vim use the same trick?
What's needed is a transaction API -- yet another good idea that was planned for someday, maybe, in Reiser4. After all, single filesystem-metadata-level operations are generally guaranteed atomic, so I would guess most filesystems are able to handle complex transactions -- we just need a way for the program to specify it.
The fragmentation issue I see as a simple tradeoff: Packing stuff tightly saves you space and gives you performance, but increases fragmentation. Running a defragger (or "repacker") every once in awhile would have been nice. Problem is, they never got one written. Common UNIX (and Mac) philosoph
Parent
Real SANs do more (Score:5, Informative)
Still, I bet you could do what you want on the cheap. Being in health care, response time and availability really are life-and-death, but many other industries don't need to spend the extra. Best of luck.
Its just not the same thing. (Score:5, Informative)
Then you have the other features like dual redundant everything: controllers, power supplies, etc. Then you have thermal capabilities of rack-mount solutions that often are different from SATA, etc, etc.
Re:Its just not the same thing. (Score:5, Informative)
We have a Sun X4500 which uses 48 500GB SATA drives and ZFS to produce about 20TB of redundant storage. The performance we have seen from this machine is amazing. We're talking hundreds of gigabytes per second and no noticeable stalling on concurrent accesses.
Google has found that SATA drives don't fail noticeably more often than SAS/SCSI drives, but even if they did, having several hot spares means it doesn't matter that much.
SATA is a great disk standard. You get a lot more bang for your buck overall.
Parent
Re:Its just not the same thing. (Score:5, Informative)
I work in the world of hardware manufacturing, and I can tell you that this "magical" more testing process simply does not exist. Hardware failures are always expensive, and we do anything we can to prevent them. To do this, we build burn in procedures based on what most call the 90% rule, but you really cannot guarantee more reliability beyond that. Better device design at that point is what will determine reliability beyond that point. Any person who says differently either does not completely understand individual test harness processes or does not understand how burn in procedures work.
In short, more money is not nessesarily better. More volume designs typically are, though...
Parent
No (Score:5, Informative)
Reliable? (Score:5, Informative)
Businesses also want service and support. They want the system to phone home when a drive starts getting errors, so a tech shows up at their door with a new drive before they even notice there are problems. They want to have highly trained tech support available 24/7 and parts available within 4 hours for as long as they own the SAN.
Finally, the performance of this solution almost certainly pales as compared to a real SAN. These are all things that a home grown solution doesn't offer. Saving 47K on a SAN is great, unless it breaks 3 years from now and your company is out of business 3 days waiting for a replacement motherboard off Ebay.
That being said, everything has a cost associated with it. If management is ok with saving actual money in the short term by giving up long term reliability and performance, then go for it. But by all means, get a rep from EMC or HP in so the decision makers completely understand what they're buying.
Re:Reliable? (Score:5, Informative)
The CMU study by Bianca Schroeder and Garth A. Gibson would suggest otherwise. In fact, there was no significant difference in replacement rates between SCSI, FC, or SATA drives.
"They want the system to phone home when a drive starts getting errors"
Of course, the other recent study by Google showed that predictive health checks may be of limited value as an indicator of impending catastrophic disk failure.
Basically, empirical research has shown that the SAN storage vendors are screwing their customers every day of the week.
"Saving 47K on a SAN is great, unless it breaks 3 years from now"
Of course, saving 47K on a SAN means you can easily triple your redundancy, still save money, and when it breaks, you have two more copies.
At the same time, the guy spending the extra 47k on an 'Enterprise Class Ultra Reliable SAN', will get the same breakage 3 years from now, he wont have been able to afford all those redundant copies, and as he examines his contract with the SAN vendor, he notes that they actually dont promise anything.
"But by all means, get a rep from EMC or HP in so the decision makers completely understand what they're buying."
Premium grade bovine manure with (fake) gold flakes?
Really, handing the decision makers several scientific papers and a few google search strings would leave them much better equipped to make a rational decision.
Having several years experience with the kind of systems you're talking about I can just say, I've experienced several situations where, if we didnt have system-level redundancy, we would have suffered not only system downtime but actual data loss on expensive 'enterprise grade' SANs. That experience, as well as the research, has left me somewhat sceptical towards the claims of SAN vendors.
Parent
No but... (Score:4, Informative)
I haven't gotten the point of standalone NAS boxes. They always were not fundamentally different from a traditional server, but with a premium price attached. I may not have seen the high-end stuff, howerver.
SAN is an entirely different situation all together. You could have ZFS implemented on top of a SAN-backed block device (though I don't know if ZFS has any provisions to make this desirable). SAN is about solid performance to a number of nodes with extreme availability in mind. Most of the time in a SAN, every hard drive would be a member of a RAID, with each drive having two paths to power and to two RAID controllers in the chassis, each RAID controller having two uplinks to either two hosts or two FC switches, and each host either having two uplinks to the two different controllers or to two FC switches. Obviously, this gets pricey for good reason which may or may not be applicable to your purposes (frequently not), but the point of most SAN situations is no single-point of failure. For simple operation of multiple nodes on a common block device, HA is used to decide which single node owns/mounts any FS at a given time. Other times, a SAN filesystem like GPFS is used to mount the block device concurrenlty among many nodes, for active-active behavior.
For the common case of 'decently' availble storage, a robust server with RAID arrays has for a long time been more appropriate for the majority of uses.
ZFS is great, but... (Score:4, Informative)
This hardly depends on ZFS... (Score:5, Informative)
That being said, this sort of solution may or may not be appropriate, depending on site needs. Sometimes support is worth it.
You're also grossly overestimating the cost of an entry-level iSCSI SAN solution. Even going with EMC, hardly the cheapest of vendors, you can pick up a 6TB solution for about $15k, not $50k. Go with a second tier vendor and you can cut that number in half.
Replace NAS? Sure. SAN? No way. (Score:4, Informative)
Cheap, redundant, and performant storage. (Score:5, Interesting)
Google have a great solution [storagemojo.com] that focuses on the “cheap” part without compromising much the latter two. If you have not read up on the Google Filesystem, definitely take the time to. At the very least, it seems to call into question the need to shell out tens of thousands for high-end storage solutions that promise reliability in proportion to the dollar.
Re:Cheap, redundant, and performant storage. (Score:4, Informative)
And hiring a team to develop something similary to google filesystem is not cheep. Even highend sans will be cheeper.
Parent
Why not good ol' trusted Linux? (Score:5, Informative)
After that you just create a filesystem on top of the raid. If you don't like ext3 or don't trust it, there is always xfs. I had some rough times with reiserfs, xfs, and ext3 and for all the experience I had I would go xfs for long running server environments (and now get flamed for this little bit, use ext3 all you want).
The advantage is that you use very well tested code.
The problem comes with hotswapping. I don't know if the drivers are up to that yet. But I also highly doubt that OpenSolaris SATA drivers for some low price chip in a low price storage box can deal with hotswapping. So Linux might be faster on that one.
That is a setup I would compare to a plug'n play SAN solution. And it totally depends on the environment. If the Linux box goes down for some reason for a couple hours/days, how much will that cost you? If it is more than twice the SAN-solution, you might just buy the SAN and if it fails just pull the disks and put them in the new one. I dunno if that would work on Linux.
Re:Everyman? (Score:4, Insightful)
I don't have a proper video camera, but I'm sure that people who do, have even bigger storage requirements.
Not only that, what with all the music you can copy of a friends HD now, your storage just jumps a bit more! (I've got literally more then 10 gigabytes of music on my desktop HD. And I know people who have hundreds of CDs, so if they ripped all those, they would have much more...)
Added to all those movies you can either rip or download...
Chuck in a decent network, family and/or friends, and you can now stream all this stuff around to wherever you want it.
I'd say then, that the most common use of all this space, multimedia. Not sure who has terrabytes of multimedia though.
Parent
Re:Everyman? (Score:4, Insightful)
80GB DVDs
120GB games
14GB/hr for DV editing
1 whole drive for OSes
RAID-5ed (1 parity drive)
So I'm up to 4 200gb drives right now, without even trying hard.
Soon I'm going to jump to 500GB drives, and I expect to be hitting their limits in a year or so.
Also, how the hell am I supposed to back up all this?! Incrementals would be 10gb+ / week
Parent
Re:Everyman? (Score:5, Informative)
Not being rich, I have a couple of external HDs totalling a little less than 1TB, and it's nearly full. The rest is archived on DVD or transfered to HD for storage (cheaper, faster and more reliable than DVD).
So yeah, I can easily imagine why any organisation dealing with huge media files would be interested. Heck, I'd be a client for a safe, multi-TB storage system if I could afford it... Not everybody only deals with text files for a living
Parent
Re:Everyman? (Score:4, Interesting)
I actually have two 48GB databases full of minimal instruction sequences for generating boolean functions. Do I win the obscure use of disk space prize?
Parent
Re:Everyman? (Score:5, Informative)
I routinely work with files that are 100 GB - 300 GB each.
Just copying one file from drive to drive takes hours.
I have about 4 Terabytes in use, with another 4 Terabytes for backup.
My usage is the exact opposite of database usage (which most storage is optimized for).
I need to copy huge sequential files. I rarely need many small reads or writes.
Because of the long times it takes to move these files around, I think NFS or CIFS would be too slow. That's why I am interested in the ability of ZFS to easily export iSCSI targets. Some tests I read showed that ZFS exporting iSCSI is about 4 times faster than ZFS exporting NFS or CIFS.
I am comparing to drives directly attached via eSATA, so it's got to be fast to come anywhere close to what I get with eSATA.
Parent
Re:Everyman? (Score:5, Interesting)
Here's why:
1) Snapshots. ZFS lets me make lots of snapshots to protect myself from user error, viruses, etc destroying my data. ZFS snapshots are so lightweight that I can make them hourly at nearly no cost in time and disk space.
2) Data integrity. Even RAID-5 can allow some errors to creep into my data (google: bit rot). ZFS has a much higher level of data integrity protection.
3) Cost/Performance. ZFS RAID-Z appears to be much faster than software RAID-5. it appears to be even or faster than hardware RAID-5. Hardware RAID-5 is much more expensive than software.
Parent