Building a Massive Single Volume Storage Solution?

An anonymous reader asks: "I've been asked to build a massive storage solution to scale from an initial threshold of 25TB to 1PB, primarily on commodity hardware and software. Based on my past experience and research, the commercial offerings for such a solution becomes cost prohibitive, and the budget for the solution is fairly small. Some the technologies that I've been scoping out are iSCSI, AoE and plain clustered/grid computers with JBOD (just a bunch of disks). Personally I'm more inclined on a grid cluster with 1GB interface where each node will have about 1-2TB of disk space and each node is based on a 'low' power consumption architecture. Next issue to tackle is finding a file system that could span across all the nodes and yet appear as a single volume to the application servers. At this point data redundancy is not a priority, however it will have to be addressed. My research has not yielded any viable open source alternative (unless Google releases GoogleFS) and I've researched into Lustre, xFS and PVFS. There some interesting commercial products such as the File Director from NeoPath Networks and a few others; however the cost is astronomical. I would like to know if any Slashdot readers have any experience in build out such a solution? Any help/idea(s) would be greatly appreciated!"

gmail

[Adult film producer]

register a few thousand gmail accounts and write the interface that will make writing of data to gmail inboxes invisible to the app.

Go the Easy Route

[Evil W1zard]

I know a certain recent Zombie network that was discovered which collectively had quite a few Pbs of storage... Of course I wouldn't recommend going down that road as it leads to you know ... jail.

Wow

[DingerX]

I never thought I'd see the day when sites were boasting a petabyte of porn.
That's over 3 million hours of .avis -- if you sat down and watched them end-to-end, you'd have 348 years of "backdoor sliders", "dribblers to short", "pop flies", and "long balls". We live in an enlightened age.

Re:I just have to ask...

[temojen]

With a project this large, they may be able to do it in-house and still take advantage of economies of scale. They can buy HDDs, motherboards, rackmount cases, etc. by the pallet or container load and temporarily up-hire some of their part-timers to do the assembly.

With a network bootable bios, the nodes could just be plugged in and install an image off a server, then customize it based on their MAC.

Re:gmail

[Stuart Gibson]

That would have been my second answer.

The first, and presumably the reason this was posted to /. is simple...

Imagine a Beowolf cluster...

Stuart

Re:PetaBox

[sycodon]

Just don't call it PetaFile.

Re:Scale

[c_woolley]

I concur. Everything I have researched matches what you have stated. It is not likely this will be a very easy task to perform on a budget (depending on what he is calling a "budget"). I would guess that GoogleFS is the only viable solution other than Lustre, depending on what he is attempting to use this storage for. If large file storage is what he desires, this may be the answer once it is released to the public.

Re:Petabox....

[timeOday]

Then I suggest using *nothing*. It's free, and will work with the appropriate hardware and software add-ons.

Re:Wow

[spuke4000]

I'm not really sure I need 348 years of porn. I usually find porn really interesting for the first 3 minutes or so, then for some reason it's not so interesting anymore. But maybe that's just me.

Re:Do It Right

[greg_barton]

Ask Slashdot answers have somehow failed to put EMC, IBM, HP, etc. out of business.

Yes, that's right... You can't do it yourself... There's never an open source solution... You must buy an expensive solution.... From a big company.... You're getting very....very...sleepy...

MMMMWWWUUUUUUHAHAHAHAHAHAHAHAHAHAHAHAHAHA!!!!!!!!! !!!!!!!!!!

Re:No Redundancy?

[Anonymous Coward]

I can think of lots of data that falls into that category. Any inherently lossy high-volume data would tollerate the loss of a few more percent as disks fail.

• Archives of Spams, for developing a smarter spam filters.
• Archives of the internet.
• Sensor data from wide-networks of sensors
• Archives of consumer info (addresses, demographics, etc) used by spam companies on low-margin viagra-ad-spamming.
  • But most probably
    • large porn archives.
    I have about 2.5TB of it here; and I'm an amateur. If I were a porn business, I could easily desire acquiring 20X what I have. And in those quantities, 90+% of it isn't worth backing up.

Re:No Redundancy?

[digidave]

"I can't imagine having that much data I didn't care about."

Hollywood script archive.

Re:Wow

[rco3]

Three minutes? You wish!

Come to think of it, so do I.

Ask Slashdot Formula:

[jlarocco]

Dear Slashdot,
I have been tasked with (insert very difficult, very important job). This is very important to my company. I have (insert number much lower than it should be) dollars to do this. I do not want to use (insert company name specializing in this exact thing) because management thinks they are too expensive. I think I can do this (insert better/faster/cheaper/...) than said company, even though they have vastly more experience and have invested much more time and research than I have. My continued and future employment probably rests on this project. Please advise.

Re:Hell, BUY it from EMC!

[lukewarmfusion]

And you'll probably get at least one nice lunch out of the sales deal. I recommend saving your lunch money and asking for sales visits from all of the major players.

I'm no storage expert but...

[Mars Ultor]

Why not store the data randomly in a dilithium matrix with asynchronous data transfer and AJAX? Maybe some RUBY on RAILS too - I hear that's hot right now. Of course, you'd have to make use of a couple of Heisenberg compensators configured in parallel to keep track account for any memory addressing issues, but no need to state the obvious there.

The SCIENTIFIC Answer

[MightyMartian]

We at Vap-o-tech 2003 Inc. (not associated with Vap-o-tech 2001 Inc. which has closed its doors due to allegations of investor fraud) have developed ToastFS 2003. Using patented CRUMB technology and high capacity BUTTER read/write caching, we are able to turn your average loaf of Wunderbread into a 200gb storage media. Simply buy a loaf of our own specially tested Wunderbread ($250 USD) along with a USB-to-Popup Toaster interface (don't worry, USB 2.0 is more than capable of handling 120amp wall sockets without a problem, except in California). Then take our Vap-o-bake ToastFS drive and pop two pieces in. For doubled capacity, buy our Vap-o-bake ToastFSx2 drive, which takes four pieces. From a command prompt, simply type FORMAT C: and answer yes. Your new ToastFS drive will be formatted in minutes. Please note that we have 24 hour technical support via 1-900-842-8524 ext 241. Please don't hang up. Our operators in the Dutch Antilles are very busy and could take upwards of an hour to get to you.

Re:No Redundancy?

[ichigo 2.0]

Maybe he needs it for a swap file? I heard Microsoft upped the memory requirements in the next version of windows.

Ask Slashdot Formula: Outsourcing!

[Anonymous Coward]

Dear Poster.

This is Slashdot India and Slashdot China. With over a Billion people combined we can do it by using PeopleRAID, which is hot-swappable once every generation, and redundant once we eliminate our "one child per a couple" restrictions, and increase our Viagra imports.

Re:Ask Slashdot Formula:

[lakin]

Dear Sir,

Use Linux.

Regards,
Slashdot

Re:Oracle, also

[menkhaura]

what on Earth kind of data do they anticipate will take a petabyte of contiguous storage?

I know. They don't know I know, but I do. It's data gathered by the black helicopters, by Echelon, by Carnivore, by our very own printers, by RFID, about every movement of every single one of us... *They* do it. They.

Re:Oracle, also

[bezgin]

Wow! This is a real conspiracy theory. All I could think of was Porn. :)

Re:No Redundancy?

[Anonymous Coward]

Ah, but the grandparent post said no redundancy.

Re:I just have to ask...

[richie2000]

Yeah, but apart from that, did it work out for you? Don't hold back, I can take the truth.