Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Hardware

Recommendations On Supercomputing Hardware? 32

Posted by Cliff from the best-bang-for-the-supercomputing-buck dept.
dameon asks: "I have been asked by my supervisor to select a replacement for our current SGI Onyx2 space heater. The current setup contains 24-195 Mhz IP27 processors, 12GB main memory, and around 140 GB of total storage space. We use it to run a bunch of CFD (computational fluid dynamics) code. Currently the demand on our system is so much that the jobs are backing up. So, they came to me with two quotes and said: "Which one is better?" I have had limited experience in the field of powerhouse number-crunchers. The two quotes I have received are from HP and SGI. SGI's quote is for: an Origin 3400 with 12 GB Memory, 24-400MHz/8MB R12K's, and 1/2 TB of storage space. HP is offering 3 9000 series N-4000's adding up to about the same specs in total, with the exception of the processors. Hp is offering 550 MHz PA8600's (1.5MB) processors in their setup (it also has more storage space setup with a hyperfabric configuration). All of the software we use will run on both platforms. So, I would like to put this to the Slashdot community: Which one is better?"

"The HP system is freaky expensive, but is the extra 150 MHz/processor worth the extra money? What else do I need to take into consideration? SGI's processors (while slower) have more cache. Overall, what do I need to look out for when spending this much money? What is the best deal? Am I missing another possible solution altogether? And yes, I already suggested a cluster of linux boxes similar to the one at Los Alamos, but the apps we use have no Linux support."
This discussion has been archived. No new comments can be posted.

CFD Supercomputing Recommendations? More

CFD Supercomputing Recommendations?

Comments Filter:
  • That's the only way to know for sure which one's faster.

    In my experience, SGI has a better HPC software development environment (read: Fortran compiler) than HP, but then again I haven't touched an HP system since we decommissioned our Exemplar 2.5 years ago.

    --Troy
  • ...then perhaps do a trial run on another customer's hardware? Does either vendor have any demo hardware? They must! Get HP and/or SGI to arrange for you to run a benchmark processing job on an equivalent system. As tolan said, on a purchase this size they should at least offer to run your code and *show*you*how*well*it*really*works.

    Definitely pursue competitive bids from IBM and Compaq.

    Good judgement comes from experience, and experience comes from bad judgement.
  • Firstly, and on-topic, I'll agree with you about the site experience with one particular variant of Unix. Particularly if most of his users are coders or otherwise serious "Power Users". (Now, in the case of fluid dynamics simulations on this sort of hardware, that's probably NOT an abuse of that horrendous term)

    Secondly, and straying off-topic, I have absolutely NO experience in Irix, but I currently admin about 40 *nix machines (mostly HP and Sun), and I have found HP-UX to be quite reasonable in the admin department. I would be very interested to know what it is that makes Irix nicer to admin. Seriously. I have heard the same said about most of the major Unix variants at one time or another, but never with any justification. Is it just that some people are more familiar with one (as I am with HP-UX) or is there really a big difference?

  • There are few machines that can match an sgi for capability to move data around. They have huge bandwidth and work very well as a system. I also understand that the new 3000 series from sgi is even crazier from a bandwidth standpoint.

    IBM and HP have very fast processors but are also very expensive. Generally not worth it unless you have a good reason for choosing them.

    Especially if you are using that kind of memory I would recomend the SGI. It is cheaper. The processors may not be the fasters but they aren't slouches either. If they want to upgrade the machine there is a very clear upgrade path (just plug in more components)
  • Go with the Orgin. HP-UX isn't as nice to admin. Some pluses on the Orgin:
    Back-plane architecture is super fast. It's the I/O champ of big computers.
    Nicer admin tools
    Your site people already knows IRIX

    I'd consider offerings from Sun, but if the only choices are SGI or HP - it's SGI hands down.

    Fast RAIDs and SANs: http://www.datadirectnet.com/ (though they don't have any SANs for SGI yet)
  • SGI Origins and Compaq AlphaServer GSes aren't even true shared memory boxes (they are distributed shared memory, NUMA).

    Heck, if you want shared memory supercomputing power, get the real stuff, either a NEC, a Fujitsu or a Hitachi (Crays are now way behind in performance for their shared memory offerings, they only have NUMA T3s).

    For many (real) applications, a NEC SX-5 with 16 CPUs will kick an AlphaServer GS 32 CPU box right out of the water. Don't look at the Top 500 (they're near the top anyway), their benchmark is so parallel that a distributed.net-style effort could get the top spot...

    http://www.hstc.necsyl.com/ [necsyl.com]

    --
    Pierre Phaneuf

  • Forgot the disclaimer (I work for NEC), but I guess that was pretty obvious from my signature... :-)

    About some talks about memory bandwidth: the data bus is 256 BYTES wide. I repeat: this is not "bits", but "BYTES"!

    There is also a crossbar interconnect device that can connect a number of SX-5s together at 8 or 16 gigabytes per second (I think it's 16, but I'm not sure).

    E-mail me if you want a sales contact.

    --
    Pierre Phaneuf

  • My Astro/CFD benchmarks show that Alpha is WAAAY faster than SGI. In fact even my sub $2k Athlon 700 home linux machine is heaps quicker than a SGI/R12k/300/8M cpu. I don't expect that HP is that much better.

    Hence get your CFD vendor to port their code to Linux/x86 and buy yourself a stack of Athlon (thunderbird) 950's. These are nearly as quick as the fastest Alphas (ev67/667/8M)... For the money you'll get about 4 times the performance out of your purchase if you can go with Athlons... If your CFD vendor won't port then find another vendor!

    If you have to go with the expensive hardware then look on the SPEC website (www.spec.org) to see how they compare - SPEC is essentially biased towards the scientific and CFD apps you are running. Look for details on each part of SPEC to see which is closest. Also look at Compaq gs320 hardware - it might be the fastest of the 'big iron'

    If Athlons came in duals or quads they'd be the only choice. We're still forced to look at Intels (which are about 30% slower per MHz and cost more than Athlons) just 'cos they come in duals :-/

    And once again, try your app on the real hardware before handing over that sort of cash.
  • Ahem, if you'd done your benchmarks then you would know that Linux/x86 ARE better. Certainly from a price/performance point of view, and ALMOST from an ultimate performance point of view as well.

    This assumes a) fast Athlons (not necessarily duals) b) distributed memory (aka MPI) apps are ok.

    Linux and gcc aren't superior to proprietary OS's (such as IRIX, Tru64 on SGI and Compaq/Alpha hardware) or their excellent C compilers, but it doesn't matter 'cos MIPS, Alpha, SPARC and HP MHz just haven't kept up with the x86 crowd.
    Of course if you need large shared mem boxes then SGI and Compaq et al are your only choice, and boy do you pay for it...
  • oops, I forgot to add --->

    ... and before anyone says that a stack'o'smaller boxes won't work - the guy said that his CFD app wouldn't run on linux, NOT that it required a large shared mem machine. Hence distributed mem systems may be an option. Even a pile of es40 Alphas would give you better price/performance than a single big shared mem box... not anywhere near the price/performance of linux/athlon, but hey.
  • The SGI is the way to go. It is easier to admin for sure. Still, take some SGI training from them. The architecture is the reason. Bandwidth between processors and memory is unsurpassed short of a Cray. Note, SGI does not own Cray anymore.

    If you are curious about the Sun, you can spec & price one yourself on their website. You would want to look at maybe a 32 processor E10000. The apps *may* run on it, but they are usually used as database servers. The E10000 architecture is from Cray. Sun bought the E10000 product line when SGI bought Cray Research Inc since the E10000 competed in the same market at Origins. E10000's sell better than Origins though.
  • wrong!

    Compaq has several Alpha [compaq.com] options.

    (I don't work for Compaq, but I use a big pile of DS10L [compaq.com]'s.)

  • Others have mentioned the Alpha option - I wanted to add to that recommendation. Single-proc performance of recent Alphas are 2-3x the performance of the fastest SGI procs.

    If you need shared-memory, you'll have to pay the bucks and get one of Compaq's larger systems and run Tru64Unix.

    If you can do smaller or distributed-memory (MPI) jobs, get a number of the 1-space DS10L [compaq.com] and run either Tru64Unix or Linux [compaq.com]. The other great thing about going Alpha/Linux is that Compaq has ported their excellent compilers [compaq.com], so you don't have to give up performance by going with Linux.

    (Yes, I know you can run Tru64U-compiled executables on Alpha/Linux by copying the appropriate libs, but strictly-speaking it's a violation of the Tru64U license. Please let's not get into a discussion of how this is a great arg for open source solutions, etc. I agree. Run Linux on your Alpha, use gcc if that's good enough, buy Compaq's compilers if you need performance.)

  • It really depends on your codes. If you're set up to use the shared memory then you should continue in that direction. If your codes are really distributed memory parallel running on a shared memory machine, then you should talk to SGI about getting a T3E. There is actually some shared memory available there, too (SHMEM is supported on the Origin and the T3E). If you are at all memory bandwidth limited then you should avoid the Origin at all costs. Our (heavily-optimized) code's performance goes in half when we go beyond half of the machine on an O2K! I.e. when you start to use the second processor on the same board (2 procs share a pool of memory on each proc board in the O2K, but you knew that). If you haven't seen this already on the O2K then you will probably not see it on the newer Origin either .

    Most importantly don't get a machine to which you can't port your code reasonably. BTW, why the hell can't you run your codes under Linux? Is this a commercial app like Fluent, FIDAP, Inca, etc?
  • I would have sent this via email, but (no-email).....

    Not to be to snide, but your employer does heavy-duty CFD in MATLAB?!?!?!?!. I hope these scripts are calling C or Fortran somewhere.

    Seriously though, what kind of performance are you (they) getting out of MATLAB?
  • well, the thing is...we have an SGI admin that really does not know much about the machine. My corporation has a lot more administrative support for HP-UX, but then they take root, and give it to no one else (bastards). So we would be in charge of administering the machine ourselves and not letting the rest of the corporation touch it. The difference in my knowledge of support for IRIX/HP-UX is really a wash (very limited in either case). I kind of adopted this program, and my predecessor had only gotten quotes from SGI and HP. I will look into the suns, but i do not know if the apps (Vectis, ABAQUS, Fluent, StarCD, ProMechanica, and others) will be supported under that platform or even a linux cluster.
  • We run apps like Fluent, StarCD, ABAQUS, Vectis, and ProMechanica. I would love to save my company butt-loads of money and set up a reasonably small cluster of Alpha-based linux boxes. But I do not think that these applications support linux yet.
  • Get benchmarks from each company on one or two of the applications you run. You'd be surprised how willing they'd be to do this. - antoine
  • HP is one of the better unixes to administer, with SAM et all helping out with the standard admin stuff.

    However the champion system for ease of administration is IBMs AIX. The with the "smitty" tool, you can add in a few new disks, and set up some a couple of mirrored file systems (optimized for database IO performance) in about 10 minutes. Compared with about 45 minutes on HP, and, days of manual reading on Solaris.

  • The CFD and the Matlab comments should be considered separate. We use Fluent for CFD, but the only benchmark software I can run on both PC's and different vendor workstations is Matlab. I was trying (poorly evidently) to get the point across that Mhz is a somewhat meaningless measure of performance. When we evaluated 4 vendors machines, the SGI's actually had the lowest clock cycle, but were only beat out by the Alphas. Actually, we had to hunt around and optimize the code for the Alphas before we could get them to beat the SGI's. With the SGI's we just compiled with -O2 and away they went. So back to my Matlab reference. Working with rockets we have some rather large scripts running double precision math, and again the SGI's just sing right along. We have a multitude of multi-processor and single-processor SGI's and they have all worked hard and fast. We are starting to get more Suns in house, and the NEWER ones are holding up pretty good too. Very solid memory architecture as well as a rock solid threaded kernal. The HP's that I have tested and our company has evaluated have not been up to snuff with the SGI's and Suns. I wish I had more eval time on the Alpha's since I believe they would also be up to the task, but alas I don't.
  • Why all you need is a Playstation 2, of course.
  • I know you mentioned you already have access to a beowulf type cluster ("like Los Alamos", but they're really *everywhere* now, and LANL is not even doing the most interesting work on them), but that your CFD code won't run on linux.

    This needs a little more looking into: why won't it run on linux? I assume it's Fortran (or C, in which case you're super lucky). What code compiles on IRIX and HP-UX, of all things, but doesn't compile under linux? Have you tried proprietary (eg PGI) compilers? Is this a library issue? If I were you I'd look into that angle a little more.

    Why would I pick a beowulf?
    - transparency: no proprietary clustering, SMP or other code. You will need to use either home-coked or open source tools to administer it (ANL's chiba tools being an example of the type), but that is much more transparent than either HP's or IBM's ways.

    - Linux: if you have a problem, you have a huge base of users and admins to talk to (ExtremeLinux folks, Beowulf underground folks etc etc). Open source makes problems ultimately the admin's responsibility, and you might not like that accountability, of course. But if you've dealt with large company support, you'd know it's not the best thing to depend on.

    - Cost: reasonable beowulfing hardware costs a small fraction of what O2k's, SP's or that class of machine cost.

    - Reusability/Extensibility: PCs are easy to downgrade to desktops after a while, just by adding a good video and sound card.

    - Scalability: first off, I know from personally having administered beowulfs larger than 24 boxes, that there are next to no scalability issues with that order of number of machines. On the other hand, you can add on as you go. Beowulfs typically perform well in inhomogeneous environments.

    I'd go with Alphas, since CFD + a lot of other scientific computing applications in physics and chem are fp intensive.

    Hope this is somewhat comprehensive.

    Btw, everyone's doing this: it's not risky anymore. On the other hand the beowulf approach has its merits, it's not a fad. That is to say, I think linux clusters, as opposed to SPs are here to stay.

    --dubido
  • Seriously, on a purchast this size cant you borrow [or even rent] the system, and test your apps on it? I realise that there is a huge amount of setting up to do, but you could just do enough to try out the systen

    Are you going to keep the present system running in parallel? if not what do you plan to do with it?
    If your system is only just starting to back up, and the new [sgi] system represents at least a doubling of power, then with both SGI systems running you'd have 3+ times the power, is that enough for the forseeable future?
    Surely IBM and Compaq (ie DEC) are worth getting a quote from before you commit?
  • Thanks for the info.

    Yes, SAM is nice, and yes, most things on solaris do take days of manual deciphering to accomplish. However, I have heard (totally unsubstantiated, mind you) that, with its reliance on database configs rather than text files, AIX is quite difficult to admin w/o the aid of smitty or Xsmit(smitX?). That's my main beef. Working in a highly heterogeneous environment, I cannot afford to get too comfortable with "flavour specific" tools, lest I forget how to do it all at the command line.

    That's why I like to install the GNU utils on every machine I admin.

    Unix ain't Unix, but GNU *IS* Unix... sorry, RMS!

    (Hey, I like that. Meet my new .sig :)

  • Have you considered a NEC SX-5 or one of their variant? Even smaller models will shove an Origin right into the ground, and many commercial applications like ABAQUS are available for these.

    NEC is involved in some Linux strategy, even at the level of its supercomputers. For example, the SX-5 can use an Intel-based Linux machine as a support system.

    --
    Pierre Phaneuf

  • Maybe I'm way off base here, I usually am. But I think with this kind of hardware it's the same as with PC and Mac hardware, you can't judge a devices ability by the measure of its clock speed, buss speed, or cache size.

    All of those things factor into how many instructions per second it can handle but you can't judge a device by that either. Little things like getting the data in and out need to be considered too.

    The software you run might run better with one hardware setup than another too.

    I don't think that you're going to know which one is better until you try each computer for yourself. Crunch the same numbers on each and see for yourself which one you're more happy with.

    You wouldn't buy a car without a test drive, you wouldn't buy a house without a walkthru. Don't buy a computer until you've cunched some data. With the kind of money your company will be spending for the darn thing you're going to have to live with what you get for a while, be damn sure you're happy with it's performance.

  • I'd also recommend you get quotes from other vendors as well. I know IBM makes some serious hardware (SP2, the computer which beat Kasparov in chess...) for uses like this. Also, they announced a new model yesterday (S85), but it might not be suited for your needs. Go check out their site (www.rs6000.ibm.com) and contact your local IBM salesveasel. Also Compaq's Unixes might be good to check out. Someone mentioned Sun, but AFAIK they're more suited for ISP/ASP use, correct me if I've missed something?
  • As a confirmed alpha bigot, I can confirm that I was pleasantly surprised by the IBM SP nodes. BUT, only the performance on your code matters. I still think for me that an alpha linux cluster is the most cost effective way to go; with the SP you are really paying for 1) the high speed interconnect 2) The fact that the software works nicely together. Other than that, it is just another cluster. If you haven't yet figured out the difference between clusters, uniform access shared memory machines, and non-uniform shared memory machines I recommend reading the first few chapters of "In search of clusters" by Pfister
  • I'd definitely check out other vendors as well - IBM has a benchmark center for the AS/400 - I'm sure there is a similar deal for the RS/6k line (an SP would be really fun). See what all of the companies (HP, SGI, IBM) will do for you in terms of benchtesting *your* stuff on their systems. I would think they would be more than happy to help.

    #include<std_disclaim.h> (Big Blue)
    --
  • SGI and HP may be the only vendors with shared memory machines this big. Certainly the IBM's are all distributed memory, except for their continuation of the Sequent line that they bought. Still, for interprocessor communication and memory accesses, nobody can touch SGI's NUMA on massive shared memory machines, whether your code is threaded or MPI.

    I think you'll find that the 400MHz R12k's (of which we have 8 in an Origin 2000) kick serious ass. The huge cache and fat pipelines beat 800MHz Pentiums by a factor of at least four in our cursory benchmarks on some of our non-parallel codes.

    Bingo Foo

    ---

  • I am working for a small company with needs for reasonable computing power and we just purchased a top-of-the-line HP workstation.

    I am more of a hardware guy. This machine is very well built; not like the cheap 712/60's I've used in the past. The electrical and mechanical construction of this computer reminds me of the big iron machines of the past.

    Plus, it's faster than hell.

    So, from a hardware standpoint I love HP; it is sweet. But as others have remarked, HPUX is a pain to admin--at least for a HPUX newbie. I have admin'd Linux and BSD boxen for seven years, but HPUX is a new animal. If it ever gets even a tiny bit confused, it throws it's hands up in the air and vomits. This annoys me.

    As far as SGI systems go, they have never struck me as very well built, even though I like them. A few years ago the University I was attending (getting a MS in EE) purchased what was then the fastest graphics computer in the world, a many (128?) processor SGI/Cray in nine racks. The dust level in our machine room (which was pretty high by computer room standards--it was installed by Univac for the 1108) caused it to fail within a month. Even with the dust problems fixed (the FS guys took the entire thing apart and vacuumed each board--plus replaced a lot) the machine has been a constant battle, and Irix issues cause it to reboot frequently.

    An 1986 vintage HP9000 series 800 mini (the fastest box in '86) has been running nonstop in the same room for going on fifteen years with almost no downtime. Yeah, it's old and is more dust-tolerant, but it also seems better built.

    Clearly I'm not an expert, but I would vote for the HP hardware over the SGI. Too bad you can't run your apps on the HP box with Linux installed...

  • We like SGI's (and Suns) (Score:3)

    by jayteedee ( 211241 ) on Tuesday October 03, 2000 @04:20PM (#733636)
    We do some heavy CFD work at my employeer too (Orbital Science Corporation). In the past, we have stuck with the SGI's after getting samples from each manufacturer and reviewing all of the offerings. SGI was bested only by the DEC Alpha machines (nothing like raw clock cycles), but the future (2 years ago) looked a little bleak for the Alphas, so we went with SGI's. They have held up well, and bely their meager clock cycles. SGI systems are always tuned to work as a system, and with CFD you are going to need it. High speed access to memory, cache, etc. Given the specs you mentioned, I would think the performance would be a wash (my 'educated' guess), so I would definately go with the lower price. I can't imagine you could go wrong. The newer PC on my desk is technically spec'd out to be twice as fast on a Mhz basis, but the SGI will absolutely clobber the PC on large computationally heavy Matlab scripts run on both machines. We are starting to comtemplate our next upgrade, and are leaning towards the UltraSparc III's. We have some beefy II's around here and they are doing real well. I'd at least look at Sun's offerings.

Slashdot Top Deals

Scientists will study your brain to learn more about your distant cousin, Man.

Close