How Well Does Windows Cluster? 665
cascadefx asks: "I work for a mid-sized mid-western university. One of our departments has started up a small Beowulf cluster research project that he hopes to grow over time. At the moment, the thing is incredibly weak... but it is running on old hardware and is basically used for dog and pony shows to get more funding and hopefully donations of higher-end systems. It runs Linux and works, it is just not anything to write home about. Here's the problem: my understanding is that an MS rep asked what it would take to get them to switch to a Microsoft cluster. Is this possible? Are there MS clusters that do what Beowulf clusters are capable of? I thought MS clusters were for load balancing, not computation... which is the hoped-for goal of this project. Can the Slashdot crowd offer some advice? If there are MS clusters, comparisons of the capabilities would be welcome." One has to only go as far as Microsoft's site to see its current attempt at clustering, but what is the real story. Have any of you had a chance to pit a Linux Beowulf cluster against one from Microsoft? How did they compare?
Licensing (Score:3, Informative)
Software costs for 100 linux machines are close to nill.
Software costs for 100 Windows machines probably won't be.
Granted I have read the licensing on the MS Clustering link but if it like anything else you'll need either a license of some kind on every machine.
Take a look at (Score:5, Informative)
Re:Licensing (Score:5, Informative)
From reading the MS Site it looks licensing is based of the EULA of the software being used, so if you are using win2kpro you have to have a copy of win2kpro for each machine etc etc.
Re:first post - no way (Score:4, Informative)
From Microsoft's site: "The Computational Clustering Technical Preview (CCTP) toolkit is used for creating and evaluating computational clusters built on the Windows® 2000 operating system."
Obviously, they are now attempting to compete with projects like Beowulf. It's probably all part of the M$ aggressive stance on Linux (and other competitors). The real question is, has anybody downloaded this kit and played with it. It's just a technology preview, so how mature is it in comparison to Beowulf or other clustering technologies?
Here's the deal: (Score:5, Informative)
Simply put, it works well (but the cost is often an issue due to the cost of hardware in an enterprise) but it is not the same clustering you see with the Unices. E-mail me at my account if you have more specific questions.
My intent is not to start or participate in a flame war, but the term clustering simply implies different things on different OS'.
MS Cluster is not the same (Score:5, Informative)
We run a MS cluster here. VERY big app... so big, I am loathe to name figures, because that would identify to MS just who is talking here...
But, we use MS clustering for our web app. Our setup is that we have a database server with 4 procs, and a growing array of web servers with 1 proc each, all of which use disk space on a SAN. W2K clustering manages the load balancing as well as allocating disk space out of the SAN to virtual partitions as needed. The original poster is correct; MS clustering is for load balancing, not computation. I have seen many times Microsoft sales reps don't have a clue of what they're trying to sell; they're just told from on high to replace Linux with Microsoft wherever they can. I think this is clearly a case of that.
My advice? Ask the sales rep to demonstrate how MS clustering will solve a common comp-sci problem with more MIPS than each box alone has. Point out that you're not running a web server or any such service on these boxes, but that they're for raw computation. Even better, see if he'll let you talk to a technician on how W2K clustering can meet your 'unique' (at least to MS) needs.
Now, for everyone else... Don't get me wrong. W2K clustering is a great technology for building highly performant, highly reliable, highly scalable applications quickly and easily. But it scales in the direction of millions of users, not millions of computations.
Windows 2000 Advanced Server (Score:2, Informative)
Distributed computing for Windows has been around for a while though, Seti@home has been doing it for years.
Stability issues (Score:5, Informative)
In terms of performance, Windows kernels have pretty good latency compared to 2.2.x linux kernels, so running a full screen dos app might give very good performance, but there's a lot of overhead munching into your RAM, which is likely to be an expensive premium on older hardware.
Lastly, with Windows, I've never heard of doing channel bonding for ethernet (3 100TX cards ~= 1 gigabit), nor diskless booting that I know of. These can be really necessary for large clusters to keep maintenance down and performance up without buying higher end equipment.
First Hand Info (Score:4, Informative)
Notes from experience:
1) Clustering with Windows requires one of the following OS setups: Win2K Server WITH MS Application Center, OR Win2k Advanced Server. (Similarly with the XP platform)
2) OS Licenses therefor will run between $1000-2000 _per-machine_!
3) If you need Application center, which you likely will, you're talking (If I remember correctly) about another $1g per.
4) Of course MS is just getting into this so don't expect it to be easy, well documented or stable.
Finishing Notes:
Obviously, Linux would be mucho cheaper
Easiest, and still cheaper than MS would be the Plug-n-Play Mac solution!
Re:MS Cluster is not the same (Score:5, Informative)
MS Computational Clustering [microsoft.com]
The OS doesn't matter - tools do (Score:5, Informative)
For a computational cluster, the OS itself shouldn't really matter. What matters is, do you have the tools you need, and does the environment allow you to work with the cluster in a flexible way.
For a typical compuatational cluster, what determines the performance will be the quality of your application. Only if you pick an OS with some extremely poor basic functionality (like, horribly slow networking), will the OS have an impact on performance.
People optimize how their application is parallelized (eg. how well it scales to more nodes). The OS doesn't matter in this regard. They optimize how well the simple computational routines perform (like, optimizing an equation solver for the current CPU architecture) - again, the OS doesn't matter.
So, in this light, you might as well run your cluster on Windows instead of Linux, or MacOS, or even DOS with a TCP/IP stack (if you don't need more thatn 640K
However, there's a lot more to cluster computing than just pressing "start". You need to look at how your software performs. You need to debug software on multiple nodes concurrently. You need to do all kinds of things that requires, that your environment and your tools will allow you to work on any node of the cluster, flexibly, as if that node was the box under your desk.
And this is why people don't run MS clusters. Windows does not have proper tools for software development (*real* software development, like Fortran and C - VBScript hasn't really made it's way into anything resembling high performance (and god forbid it never will)).
Furthermore, you cannot work with 10 windows boxes concurrently, like they were all sitting under your desk. Yes, I know terminal services exist, and they're nice if you're a system administrator, but they are *far* from being usable to run debuggers and tracing tools on a larger number of nodes, interactively and concurrently.
Last but not least, there are no proper debugging and tracing tools for windows. Yes, they have a debugger, and third party vendors have debuggers too. But anyone who's been thru the drill on Linux (using strace, wc -l
So sure - for a dog&pony show, windows will perform similar to any other networked OS with regards to computational clusters. But for real-world use ? No, you need tools to work.
channel bonding (Score:3, Informative)
however, I would take issue with your assertion that 3 100mbit cards are roughly equal to a gigabit card. while it's true that something like 4 100mbit cards will give you close to the real performance of a gigabit card when used on a low end PC, there is much to be gained by using actual gigabit (use of giant frames, better latencies, etc.)
if you're going to build a cluster, and you actually have a budget, you're going to buy decent yet cheap server boxes. these will most likely include 64bit PCI slots, and there lies your motivation for gigabit. the performance there is unparalleled when using a real wirespeed switch, without using faster technologies of a proprietary nature.
my 2 cents.
Re:Point? (Score:2, Informative)
Re:Point? (Score:1, Informative)
So unless you can finish you project in what - 30days, 60 days??? then I guess this would be useless licensing.... Some projects actually take longer to finish one job then what your
FREE LICENCES
FREE DEVELOPMENT TOOLS
last... SO STOP YELLING D#MB F#CK!
MS HPC vs. Beowulf (Score:2, Informative)
MS clustering is for load balancing and stability. As such, it does a reasonable job. Beowulf is for high-end scientific computing and does a resonable job.
To really do clustering well, you don't want either. You want AIX or Solaris, but you probably can't afford them. But, Linux clustering in load balancing style is developing quickly with IBM, TurboLinux, VMWare and Intel doing many interesting things. Beowulf is still cheaper.
In any case, though, for your situation, there's only one solution and that's Beowulf. Besides afford MS' licensing fees, you mentioned that you're running on older equipment. I sincerely doubt that those servers could run W2K Server in its standalone mode, much less in clustering.
Steven
Re:Clustered MPEG encoding with TMPGenc? (Score:2, Informative)
not fun (Score:2, Informative)
use Application Server, not Clustering (Score:5, Informative)
-
Failover clustering. This is an OS service that servers like SQL Server and Exchange plug into that allows Active/Passive or Active/Active clustering over a shared SCSI/Fibre bus. In theory you could write your app to use this service but I think it would be overkill.
-
Network Load Balancing. This is just a software version of the standard kinds of NLB found in cisco boxes.
-
Component Load Balancing. This is the most suitable. It's provided by Application Center and it allows you to deploy COM+ objects on a cluster of machines and have the calls distributed according to the load on those machines. You can control the threading and lifetime of the objects and view the status of the machines pretty easily using the Application Center MMC plugin (or SNMP, I believe). You'd have to wrap the computational part of your application into one or more COM objects. Once you've done that then you can create and call those objects in the cluster as if it were one machine - the clustering is transparent to the client application. I played around with AC a bit when it was in beta for a project that I was working on. We didn't go with it in the end because the design of our application ended up not requiring it (we just went with hardware load balancing), but it seemed like pretty cool technology - if you're into the whole COM thing. It has a really cool rolling deployment feature where you can redeploy your components (and/or IIS application if you have one) to your cluster incrementally while it's still running.
Here's some links to docs on MS's site:Introducing Windows 2000 Clustering Technologies [microsoft.com]
Application Center home page [microsoft.com]
Component Load Balancing [microsoft.com]
Microsoft Clusters have been done. (Score:2, Informative)
Back when Beowulf started there were a few universities that had dual boot systems. I think they ran Linux half day, Windows the other. Windows NT and above have had implementations of MPI and PVM that have worked quite well for a while. MPICH and PVM are both freely available, the problem is that these systems do not integrate well with Visual Studio.You end up needing cygwin/gcc, which is basically Unix on Windows
Most clustering software today is either open source or based on open source and made for the Unix environment. So besides the obvious license and stability questions others have brought up Windows has limited tools and libraries.
ALSO:
There are problems with remote administration, being tied to graphical interfaces for such simple nodes. Wasted efficiency by running the GUI.
Finally Microsoft has a VERY limited knowledge base for this application of Windows nor do many HPC people know anything about Windows.
As you can see from the above, Linux on HPC is basically able to take those same horrible excuses for running Microsoft on the desktop and shove it down their throats.
Yes, Windows can do HPC, but why would you want to?
Windows 2000 Clustering (kinda) explained... (Score:3, Informative)
A few points:
You can set up any application or service to cluster & fail over if required, as long as:
Active/Active mode is more complicated, meaning instances of an application running on different nodes, all accessing the same data on disk. Only certain applications can do this successfully, e.g. Oracle, which does so by using a custom file system and effectively bypassing the Windows Cluster Service. Windows & most apps will normally throw a fit if there are clashing file requests from multiple nodes, since Windows caches file tables in memory and can thus lose track of the real situation on disk (bad news). I've seen it BSOD in such cases.
Re:MS Cluster is not the same (Score:5, Informative)
Notes for any and all interested in this; it's a technical preview, which any other company would call a pre-Beta or an Alpha release. The only way anyone sane would use this in a production system would be as an Early Adoption Partner...
Globus is a Linux project, sponsored by Microsoft (Score:1, Informative)
About that Mac solution..... (Score:4, Informative)
Yellow dog linux sells a cute little piece of hardware designed for clustering around PPC. very cute...maybe the best balance of cost effective and easy in terms of clustering that ive seen.
http://www.terrasoftsolutions.com/products/briQ
-jef
Re:MS AppCenter server (Score:2, Informative)
MS Application Center is not bad. It is web focused, but with webservices alot of things can be web systems.
Good Features.
Problems
It has advantages over the Linux systems I have see with the GUI and aggregation of preformance data. The GUI is useful because you can not delegate tasks to junior staff, if they do not understand it. I have 60 web servers many running Linux and some W2K and a few have been trying app center for a while now and need performance information to know when to add and hopefully remove machines. I have run mainframes, but there are less software problems with popular systems like intel Linux and Windows machine so zSeries and big Suns which I have used in the past our not the magic bullet.
Seán
Re:, but what the hell (Score:3, Informative)
I used to have some WLBS (Windows Load Balancing Services) systems (NT4's idea of load balancing cluster).
They worked, more or less, most of the time (about 4 reboots/day on average I think). The problem was, the thing was IMPOSSIBLE to debug and troubleshoot, for the simple reason that it was impossible to know where the problem was. WLBS did terrible layer 2 trickery to route requests around, and as a result it didn't work well with anything more complex than a hub.
Luckily it's now gone and not missed.
Disclaimer: the opinions here expressed are of course my own and do not necessarily reflect any organization's
Pooch for the Macintosh (Score:3, Informative)
This would allow you to use the Macs (OSX UNIXY goodness too) individually as personal workstations (for writing, graphics, computation, surfing the web) while at the same time using them in clusters for compute intensive work. This makes for a doubly productive machine and one that is much cheaper as more work can be accomplished with it than simply using it as a dedicated node.
Mac clusters are easy peasy to set up (even junior high students are doing it) as the one page instructions should indicate and Applescript'ability. Also pretty damn fast given the built in Gigabit of G4's and the Altivec (if taken advantage of like in Apple's version of BLAST).
Finally, the other item of interest. You can use any Mac you have. G3's, G4's of any model and speed as one does not have to balance everything like on typical clusters where all of your hardware has to be exactly alike. The Macs in your cluster can even have iMacs on the secretaries desk involved!
Re:No command line (Score:1, Informative)
Windows does have the powerful command line - (no, not command.com), its called Windows Scripting host (WSH) and runs practically any script language (though typically you stick with VBScript).
I combination with WMI (which is Windows' imlementation of WBEM), you can do far more than ksh, and it is relatively easy to use (once you know what to do).
Cheers.
Re:That's easy/ Wish I could. (Score:1, Informative)
Re:MS Cluster is not the same (Score:3, Informative)
This is a great idea. Scalapak benchmarks are a popular choice. Also think about what are you really getting for your money (license fees)? I work with a modest Beowulf (~50 cpus) using Linux and I have no doubt that it would be technically possible to use Windows... but you would spend a lot of time installing kludgy ports of unix tools: cygnus wintools, PBS, rsh, perl, etc. At the very least the two most popular message passing libraries (MPI and PVM) both rely on rsh.
All the tools that make a Beowulf what it is are free software, there is really NO added value by running them on Windows.
MSCS (Score:2, Informative)
Quad Pentium 500 MHz Pentium Xeon's
1 GB memory
6 disk array, three logical mirrored drives.
MS Windows NT Server 4.0 Enterprise Edition
MS SQL Server 7.0 Enterprise Edition.
It should be noted that you must have two NT Server licenses and two SQL Server licenses. If you want to do an active-active cluster it requires four licenses. The Enterprise Editions of these software packages was much more expensive than their standard counterparts. You can not use standard editions for clustering.
Installing cluster services was very easy. The Cluster Manager app was OK outside of the occasional hangs. (Although the manager app hung, the operations were completed, such as failover, failback.)
In order to do active-active clustering you must have two shared storage devices; the active node will only be able to access the shared storage it "owns".
SQL Server installed all right if you followed the MS White Paper exactly. I don't know why, but installation order was important; if you didn't follow it it didn't work.
Applying service packs was extremely painful. The instructions were straightforward but did not work. MS provided us with a program that backed out the SP snafu, which worked somewhat. If it weren't for google we'd have been dead.
MS support is useless IMO. No contracts just pay-by-incident. Have a credit card handy before you do any upgrades of any kind. You will most likely need it.
As long as the cluster was just doing SQL server, it worked great. Failover was seamless. Given the proper hardware, Windows behaved well. Make sure that you only attempt this with certified hardware. Very important.
Once we started adding third party reporting software things started to go bad. Adding it to the cluster services was remarkably easy. However, even though the servers had quad procs and a good amount of memory, simultaneous report requests ground the system to a halt. SQL Server behaved well, around 25% of CPU at most even in heavy load. The reports (JRE) would take up over 50% of the CPU in light load. Very bogus IMO.
A lot of third party apps do not support MS Clustering. Lot's of tweaking to get them to work.
If I were to do it again, I think I would not have used MSCS, but instead have two distinct systems that had some kind of data replication software.
This configuration is also limited to a two node cluster. Although you can run an active-active cluster the instances of SQL Server would be seperate. The data storage areas cannot be shared between the two nodes.
Although I prefer UNIX I try not to be an MS bigot. It does certain things well. I hope that clustering has improved with w2k.
Windows clusters pretty well (Score:2, Informative)
http://www.mpi-softtech.com/
A couple years back, I was one of the MPI writers/maintainers for a number of platforms (worked with MPICH). As far as performance went with real applications (as well as synthetic benchmarks), Windows and Linux on the same hardware were pretty much the same. Typically computational problems were faster on Windows though because of the better compiler support at the time. Communications performance benchmarks were interesting.
Also, the at the URL above you can find cluster management software (batch scheduling and stuff).
Re:You're running on old hardware right? (Score:2, Informative)
chavo
Re:Licensing (Score:4, Informative)
At the end, it all comes to your soft, if you develop a highly scalable, almost share nothing algorithm, Linux Clustering is the way to go. For fail-over Linux you have tha HA Linux project, once more, Open Source!
Re:Point? (Score:5, Informative)
"Q. How does a Windows-based supercluster compare with one running UNIX or Linux?
A. In short, there's very little substantive difference, but owners of existing UNIX-based solutions will face changes that will cause them some work and discomfort (less for users than for their current administrators and support staff). These are offset in part by lower costs of ownership (technical skills required), breadth of applications and support tools, vendor support options, and commonality with the constantly improving desktop environment.
From a hardware perspective, there's very little difference seen by the application. In the past, UNIX-based hardware environments had better floating-point performance, but that's been offset in the last few years by Moore's Law curves for large-volume products that have advanced faster than specialty products have, as well as the price and support cost differentials between these vendors' products.
From a software perspective, Windows is a markedly different environment, designed with priorities set by a much different market segment than traditional science and engineering. Windows NT® and now Windows 2000 were designed to meet the needs of those ISVs building products for businesses that are unable or unwilling to dedicate their best people to support their infrastructure (versus focusing on building solutions for their business mission), as well as the needs of a hardware community that required continuous integration of new devices and components."
Re:MSCS (Score:2, Informative)
And yes, you have to follow many sets of conflicting directions to a friggin' "T" or else it won't work. And do yourself a favor and firewall the hell out of the boxes. Installing service packs on such a mission critical set up just doesn't appeal to me for some reason.
By the way, you say:
"...If I were to do it again, I think I would not have used MSCS, but instead have two distinct systems that had some kind of data replication software."
I have no idea how you intend to accomplish this with a database, utilizing a MS solution, but I'd certainly like to hear about it!
MSCS Vs. Beowulf = Apples vs. Oranges (Score:3, Informative)
BSODs are an issue (Score:2, Informative)
Windows NT 3.51 had the video drivers (and most other drivers as well) in the outer ring of the kernel where they couldn't down the whole machine, just certain services. I've seen Win3.51 boxes with horribly buggy video drivers just keep right on running when the video would lock up. NT 4.0 and above aren't the same.
The decision to move the drivers into the inner ring of the kernel is why BSODs are a Windows issue. Blaming the user for not setting up his box just right doesn't solve the real issue, poor OS design.
A real OS (unix/linux/OS390/VMS/even NT3.51) doesn't have these problems.
Shameless plug (Score:3, Informative)
We have been very successful in Windows clustering efforts and offer a professional MPI implementation for windows platforms. Give us a shot I am sure we could set up an evaluation of some sort.
That said, we have the following self-kudos:
CORNELL THEORY CENTER'S VELOCITY CLUSTER MAKES THE TOP 500 LIST (June 16, 2000)
"Our relationship with MPI Software Technology, Inc. has been extremely valuable," says Cornell Theory Center associate director for systems Dave Lifka. "Good job scheduling, resource management, and reliable MPI are the primary pieces of any high performance computing environment. MSTI has made the extra effort to make sure MPI/Pro and Cluster CoNTroller are ready for a production quality environment. The utilization and stability the AC3 systems is directly related to the quality of their software."
World's Largest NT Cluster Goes Live (August 25, 1999)
The Advanced Cluster Consortium (AC3), which includes Cornell University, Intel, Microsoft, Dell, Giganet, and MPI Software Technology, Inc., announced on August 12, 1999, that it had completed the installation of a 256-processor high-performance computer cluster using Windows NT 4.0. AC3's cluster bests a University of Illinois 192-processor NT cluster, which Windows NT Magazine covered in June 1999.
As you can see we've been at it a while!
MS clustering = bad mmkay? (Score:4, Informative)
2 x HP Netservers, both dual p2 Xeon, 1gb ram, and a small raid shelf with 8x 9gb disks. Both NT4 installs with the correct patchlevels.
One machine ran oracle, the other IIS, these were clustered so that one would take over the task of the other, should there be a problem.
Problems:
1) Crashing (daily at least)
2) Slow (astonishingly poor, disk defrags once a week helped this)
3) Sometimes one host would freeze, and the other wouldn't actually notice
4) Often a shutdown of one node would move the services across, but upon rejoining the cluster - the node with both services would refuse to give one back.
5) Often, IIS would stop talking, and neither node would actually realise.
The attempted solutions:
1) Replaced CPUs, memory, disks, eventually nodes
2) Reinstalled clustering software, eventually total clean installs of operating system and applications
3) Support from Microsoft, and Oracle, and HP who made the (certified) kit. Oracle+HP both pointed the finger at the OS, Microsoft simply failed to help, when we got any response from them at all.
4) (this helped) I used one of the spare HP9000 servers to monitor them remotely by trying test transactions - it alerted people when they fucked up.
I think the above says it all really. Standard software on correct hardware - it just didn't work properly. Microsoft can stick their clustering "technologies" where the sun don't shine.
Re:channel bonding (Score:1, Informative)
Windows HPC Cluster (Score:2, Informative)
Re:BSODs are an issue (Score:2, Informative)
Yes, Microsoft puts the drivers right in the kernel, but other OS's end up with similar results anyway.
For example, Linux does have kernel level drivers(DRI) for most common graphics cards nowadays simply to increase performance and to allow for features User space cannot perform. Plus, there is a growing wave of frame buffer kernel drivers which give users space a blank frame buffer to work with, so that programs like X, gnomefb, kembedded don't have to worry about writing to every hardware platform, just the abstract one that the kernel produces, pretty much the same as what windows uses.
This may not be or ever be a standard in all UNIX's, but anything that needs performance out of graphics, or any non-core perpheral, they will be going into the kernel, or they will have special hardware.
Done.
Re:Headless Windoze machines (Score:2, Informative)
For the record, I maintain a headless NT 4.0 web/database server at work for one of my projects (requires disabling the mouse driver to avoid error messages at startup) controlled via PC Anywhere and a headless Win98SE machine at home as my internet gateway (running SyGate NAT 3.0 and SyGate Personal Firewall) controlled via VNC.
Why NT 4.0? Mandated at the time (the main servers are in the US maintained by an external group - we're in Australia with the admin server for the same system).
Why Win98SE? I tried various linux and bsd distributions on the machine, and couldn't get any to work - Pentium 60MHz, SCSI, plain IDE (not ATAPI) so had to install an I/O card to get a CD-ROM to work, old intel ethernet cards, etc. I've configured it to reboot every night so I don't have stability problems
MS parallel tools (Score:5, Informative)
COM+ and Queueing Components. AppCenter.
The way it works is this. You write a COM+ component that is transactionally queuing aware. Each component takes a work unit in, processes it, and then sends the result of the transaction to the queueing components for reassembly or re-issue (if a node fails to submit a result, for example, good for checkpointing).
You can use normal Windows 2000 Professional boxes for the worker bees, and use a few Windows 2000 Server boxes to co-ordinate the issuing of jobs and control, and munging the result sets coming back in.
If you need to submit a wide variety of jobs, obviously the COM+ components will be changing regularly, it'd be a good idea to go to AppCenter so that you can treat a bunch of machines as single whole. This allows you to upgrade or deploy an app in a few mouse clicks to literally thousands of machines in a few seconds. AppCenter also has pretty good resource management, something that might be necessary if multiple jobs are running at the same time.
The cool thing is the development environment is really friendly and you can make COM+ components pretty easily and test them locally (for the n=1 case) before deploying to the farm.
There are also specialist MP libraries for the Win32 platform, such as PVM or MPI (WMPI). These have the benefits of re-using the knowledge and API's that users might already be familiar with - one of the biggest thing when a place converts from one supercomputer to another is rejigging and reoptimizing the code for the new architecture.