Building a Linux Cluster from the Ground Up? 17
dooling asks: "How would one go about building a Linux cluster from the ground up? I read a lot about Linux clusters on /. and have been able to find some information on configuring the cluster, but have found little on how to assemble the hardware, i.e., what is necessary, how they should be connected, etc. So does anyone have reliable information on hardware assembly and configuration? Also, (if you've never done this before) is it worth building your own, or is it better to just buy one prebuilt and preconfigured? If you want specifics: 20-40 machines, Linux (probably RedHat 6.x), disk or diskless?, do not need video cards (but should we have them?), switch or hub (best way to hook them up). We will be doing pretty straightforward scientific computing (floating point number crunching). "
Re:Something to do with a Beowoulf?? (Score:1)
beowulf would not increase the speed of any of these apps since they are not coded with a multi-processor platform in mind.
-xyster
floppy drives: needed? (Score:1)
the first cluster i set up didn't have harddrives and i nfs mounted the filesystems across the network. worked pretty well but it was a pain to get working.
Some thoughts (Score:2)
I disagree in a few places (Score:2)
1. Custom-build
Definitely the way to go. You can get nice machines for a few hundrew bucks each. But put cheap video cards in -- it makes maintenance much easier. And some MB's may not boot without them. Spare ISA ones that are lying around should do the trick -- you'll never be taking them out of text mode. The machines we got had cheapie vidcards on the MB, which was fine.
2. HUB
Switched all the way.
3. Versions
If you want the latest ver, use the ones from Debian potato (unstable)
5. CPU
I recommend Celeron 450a's (300a's OC'd to 450MHz). You'll need a nice motherboard that will let you set core CPU voltages (or some Celerons may not OC, which happened to us). But there are some relatively inexpensive dual motherboards that let you set core voltage, and Celerons and slockets are still pretty cheap (our machines were before the slockets came out, so they're single CPUs).
Celerons are far faster than equivalently clocked K6's, so go with Intel unless you want to spring for K7's (now that would be slick!).
6. NICs
I'm not sure what the gigabit advantage would be -- probably depends on what you're crunching. Obviously if computation time is high relative to data quantity, you're fine. But gigabit equipment is expensive. Consider ATM equipment -- fast and cheap. ATM switches are way cheaper afaik, and there are a couple of ATM boards supported by Linux. It's ideal for this kind of application since only the head node (which would then need another NIC) needs to talk to the outside world.
8. PGCC
I've seen no indication that
a) Pentium-optimized code is particularly better (and I suspect its stability
b) It's faster at all on PPro-based chips. Optimizing for PPro and Pentium are two very different things. I'd just go with a standard Linux distro -- it'll make your life easier.
It's irrelevant anyhow, since your code will make all the difference. It might be worth playing with different compilers to see what makes your stuff go fastest. Post the results for the rest of us!
9. Overclocking
Any moron can OC a Celeron 300a to 450MHz with a decent motherboard. Beyond that takes guts and skill -- and may not be worth it, since a decently sized cluster (ours was 16 machines) will start to show some variance in chips -- as we found out. On the upside, the load balancing software should be able to compensate just fine if you have a few dud nodes. Many Beowulf clusters are heterogeneous.
Re:Some thoughts (Score:2)
While the Cisco products are damned good, they are fairly expensive. There are other 100Mbit switches out there that will work fine that are considerably cheaper.
Run fastether (preferrably 3com) cards in EVERY node
While your advice to use 100MBit NICs is correct, you might want to rethink your suggestion of brand. Look at what the people who are building big Beowulf clusters are using. Its usually Tulip chipset based cards. I don't know if more recent models are better or if the drivers have gotten better lately, but not that long ago the general consensus seemed to be that 3Com 100Mbit cards were reliable but disappointing performers. At any rate, you can get a Tulip (or PNIC) based card like the D-Link DFE-500TX, the Bay Networks NetGear FA310TX, or the LinkSys (forget the model number -- they make two 10/100 cards, one of which is a semi-NE2000 clone and should be avoided, the other uses a PNIC2 and is the one to get) for under $50, whereas the 3C905 series cards are generally $70+.
Re:Some thoughts (Score:1)
Hubs allow packet collisions, which will KILL a beowulf. Use a switch. Switches dont -always- nuke collisions, but at least more than 1 node can transmit at a time. Hubs are totally shared, so if you start getting more than 1 computer trying to transmit at once, it'll go ackbewm.
I recommend a Cisco Catalyst 100Mbit switch. Run fastether (preferrably 3com) cards in EVERY node.` Channel bond if necessary.
Having come from a 450 resident dorm with a counted 600 active computers on a flatnet (totally shared, no switches or routers), I can tell you hubs are -slow- for more than 3 or 4 systems.
Re:floppy drives: needed? (Score:1)
I know it is not in the server setup, I can still connect with the old bootdsks, but I get a failed root mount with the Slack 7.0 bootdsks.
I upgraded because MOSIX will run under Slack 7.0 with very little alterations in paths.
Re:Something to do with a Beowoulf?? (Score:1)
Try using MOSIX, it can spread the load between the various machines even if the software is not written for it. If you can convince it to run, it is very picky about kernel versions, file positions, etc.
Other Clusters? (Score:1)
What about Mosix, for instance? Does it work as well as advertised? It sounds like a very cool thing, doesn't anyone experiment with it and share their conclusions?
And what about clustering for 24/7?
No, I can't spell!
-"Run to that wall until I tell you to stop"
(tagadum,tagadum,tagadum
-"stop...."
Re:Some pointers (Score:1)
Video cards: if you want to use a KVM switch for administration, you'll need them: in that case, go for cheapo cards for the computing nodes, and get a top-of-the-line one for the front node.
I recently [linuxshowcase.com] saw VA Linux's [valinux.com] set-up for a linux cluster: their kick-ass feature is administration thru a serial console (no need for more expensive KVM switching) and direct manipulation of each node's BIOS from the front-node (no cpu cycles spent on that). OTOH, I wasnot impressed by their top specs on their 2U systems (2 PIIIs, 1 GB RAM total): some of the code I am porting needs ~1GB/CPU, and that's not a top-of-the-line code either. Plus, 2-CPU systems mean more rack space (as opposed to quads); however, SMP performance under Linux still has a long way to go, so YMMV...
A final note: be careful about disk I/O: the common solution on MPP machines is to have each process write to disk; in linux clusters, this usually means writing to some NFS-mounted partition. Well, the NFS implementation (and I am not sure here if it's NFS in general or linux's version) cannot keep up with a lot of reads/writes to the same space... Change the way you do I/O (one process handles everything) or clone the filesystems and use local space (a bitch to maintain, though)...
Beowulfs are cool, but we are not nearly commercial-level usability here... what did the old maps say, "monsters here"?
Some pointers (Score:2)
Also, if you're not gonna need some of the more specific software (kernel patches, ethernet channel bonding and the like that usually come in RPMs) but just want to implement generic MPI or PVM, I'd go with Debian next time instead of RH6.0, purely for maintenance reasons.
The big question you have to ask yourself though, is what kind of application you want implemented, and build the cluster to match it... if you don't have an application in mind, then you probably don't need a Beowulf...
Here's a book that should be of help (Score:1)
How to Build a Beowulf (a guide to the implementation and application of PC clusters)
By Thomas L Sterling, John Salmon, Donald J. Becker and Daniel F. Savarese. ISBN 0-262-69218-X
From M.I.T Press
Published 1999
I've read this book and although some of it is very basic (such as how to install Linux) It is a very informative book, well referenced throughout If you are intrested in building clusters on other OS's as well.
Re:Something to do with a Beowoulf?? (Score:1)
Something to do with a Beowoulf?? (Score:1)