Can .NET Really Scale? 653
swordfish asks: "Does anyone have first hand experience with scaling .NET to support 100+ concurrent requests on a decent 2-4 CPU box with web services? I'm not talking a cluster of 10 dual CPU systems, but a single system. the obvious answer is 'buy more systems', but what if your customer says I only have 20K budgeted for the year. No matter what Slashdot readers say about buying more boxes, try telling that to your client, who can't afford anything more. I'm sure some of you will think, 'what are you smoking?' But the reality of current economics means 50K on a server for small companies is a huge investment. One could argue 5 cheap systems for 3K each could support that kind of load, but I haven't seen it, so inquiring minds want to know!"
"Ok, I've heard from different people as to whether or not .NET scales well and I've been working with it for the last 7 months. So far from what I can tell it's very tough to scale for a couple of different reasons.
- currently there isn't a mature messaging server and MSMQ is not appropriate for high load messaging platform.
- SOAP is too damn heavy weight to scale well beyond 60 concurrent requests for a single CPU 3ghz system.
- SQL Server doesn't support C# triggers or a way to embed C# applications within the database
- The through put of SQL Server is still around 200 concurrent requests for a single or dual CPU box. I've read the posts about Transaction Processing Council, but get real, who can afford to spend 6 million on a 64 CPU box?
- the clients we target are small-ish, so they can't spend more than 30-50K on a server. so where does that leave you in terms of scalability
- I've been been running benchmarks with dynamic code that does quite a bit of reflection and the performance doesn't impress me.
- I've also compared the performance of a static ASP/HTML page to webservice page and the throughput goes from 150-200 to about 10-20 on a 2.4-2.6Ghz system
- to get good through put with SQL Server you have to use async calls, but what if you have to do sync calls? From what I've seen the performance isn't great (it's ok) and I don't like the idea of setting up partitions. Sure, you can put mirrored raid on all the DB servers, but that doesn't help me if a partition goes down and the data is no longer available.
- I asked a MS SQL Server DBA about real-time replication across multiple servers and his remark was "it doesn't work, don't use it."
You've got bigger issues. (Score:5, Informative)
1. Why are you wed to C#, especially in regards to triggers? How many tiers exist, and are you pumping a lot of data back-and-forth.
2. Your scaling numbers are low already, especially under ASP and static HTML.
3. You never really define concurrent requests. For some people, it means simultaneous requests, and for others, it means simultaneous transactions. But you really are looking at fairly low numbers there, in either case.
4. Scaling this should involve looking at where you choke. One common choke point that keeps killing people is in open database connections. Are you running a pool? How large? How many connections does a page take? The single most common problem I've seen in scaling is poorly implemented connection pooling, thereby causing a ton of stuff to wait. Check this, check, then check again.
5. Sync versus Async shouldn't really be coming into play yet on the db.
6. When designing for light-weight systems, you want to minimize the tiers, and minimize the data passed back and forth. Just by reading this, I'm worried that you created a very elegant, but impractical, system that isn't suited to the hardware limitations.
Re:Christ, those machine figures! (Score:5, Informative)
Not even a hiccup. then the bright guy tried to do that on windows 2000 for another customer.. choked at about 100.
I'll take good performance with low spec hardware over the ability to scale on 10+ CPU systems anyday.
.NET Benchmarks (Score:5, Informative)
I have been Developing a
Specs Are as Follows:
App Server:
Duron 800
512 MB RAM
40GB HD 7200RPM
DB Server:
Celeron 500
640 MB RAM
20GB HD 7200RPM
As you can see, these are not server class machines, but they seem to run the app alright. I ran a simulation of this application based on the IBS Portal www.asp.net [asp.net] running 150 Concurrent Requests Per Second:
The average Requests per second on this app were 98.51. So, IMHO on low quaility hardware, the
Re:Any more information? (Score:3, Informative)
If that gets too goofy we may end up partitioning the requests in the beginning and mirroring two seperated complete systems, but we're not really envisioning it ever getting that big.
Of course, most of the problem is simply the back end keeping up because the front ends don't do much at all except call the java app and return the data it gets...
Ackward Worries, Threading and Responsiveness (Score:5, Informative)
Anyway. If you can't support 100 requests a second on 50k of modern hardware, you have huge design issues and other problems. Just from your short description of the project, I fear you have crawled into over-engineered land because alot of the technologies are much more useful on seperate boxes/distributed enviroments.
Good Luck. Remember that C# Web apps can be multi-threaded, and remember to optimize the parts of your application that MATTER. A wise man once said "Premature optimization is the root of all evil". Find the slow parts, fix them, get the most bang for buck. Also, remember to keep those pieces loosely-bound to each other, no C# code in the DB!
--MetaCosm
P.S. I hope you haven't over-engineered this tool as badly as it sounds like you have
This works... (Score:2, Informative)
Two cheap boxes, one running the server and the other SQL server, will outperform a single box by a wide margin. SQL Server's a pig and doesn't share well with the other children. Use back to back NICs to the connect the SQL box so there's no network overhead...
Check the check boxes when you compile your .Net components. Threading models matter. And a stateless contiuously instantiated module is the only scalable solution. Check the stats on construct/destruct overhead.
Use an n-tier architecture. Not just for the obvious reasons but because you can build faster data access including invisible data caching (as the app grows) and avoid the problems that are driving you down to only 20 or so tps.
Buy more memory - Doh!
Re:Any more information? (Score:3, Informative)
Yes, .Net can scale--IF... (Score:5, Informative)
Hi!
Executive summary:
Yes.
Boring details: .Net (mostly C#, some components in VB), including Windows forms and ASP.Net web pages. (Why both? The project incorporates multiple applications for different kinds of users.) As part of pre-shipment testing we're in the midst of extensive testing, including load testing.
I'm goofing off, perusing SlashDot at the end of a dinner break. We're shipping a big project to a customer on Monday--the project is written in
The Windows applications communicate with the data tier using SOAP/XML, using synchronous messaging. Practically every message involves a database transaction with SQL Server 2000. Across a range of loads we are seeing round-trip message responses (from receipt of the inbound XML message to return from the web service) averaging less than 90 ms per message. That 90 ms average can be misleading--some of our messages involve extensive processing and/or lots of data. Some of the transaction work we're doing with SVG images involve SOAP messages with payloads greater than 1 MB, so the average gets dragged out.
Based on our testing, we anticipate supporting hundreds of simultaneous users--in a near-real-time environment--from a single web service. As we scale out on larger projects we may need to scale the number of web servers (although IIS on Windows 2003 is supposed to be substantially faster--YMMV), but we won't need to scale the database. Using a similar messaging architecture for a different client I have a project supporting 400+ users on a single SQL Server.
This is SlashDot, after all... .Net. And recommending it. But you asked, so I'll answer: .Net is scaleable in terms of the final application, and .Net is scaleable in terms of the size of the development team that is involved. This project involves 19 developers (a total of 60+ individual projects in the nightly build) and we're able to manage the entire thing remarkably well. Developing web service applications with .Net is remarkably easy to do; developing sockets apps is unbelievably simpler than using WinInet.dll. And the web developers are extremely happy working in ASP.Net--I don't know where you heard that ASP.Net is slower than ASP, but that's simply not true. ASP.Net is significantly faster.
Obviously you're going to get a lot of "why not use...?" posts, and I'm sure I'll get flamed for having the temerity to admit to using
With regard to other comments .Net Remoting. Quick to prototype, barks in production. Like OLE, it's a great way to make a Pentium 4 box emulate an original 8086 IBM PC. (Far smarter to manage communication with XML-based messaging. It just takes more coding.)
I'm the data/messaging architect on the project: I can speak to the comments about messaging, reflection, and SQL Server. As with any Microsoft-based development project, you have to think carefully, and think critically, about how to design your application. Microsoft will always give you a quick! easy! fun! way to rapidly produce a prototype. You have to dig deeper, and think harder, to produce a scaleable application. The quick! easy! fun! technology du jour is
That SQL Server doesn't permit triggers to be written in C#--so? Transact-SQL is suitable for database development. We could ask for more (such as integrating stored procedures and other database code into Visual SourceSafe). There is talk that the next version of SQL Server will permit coding in .Net languages--that'd be cool, but I'll wait and see.
The single most compelling argument for .Net .Net Framework. You might look into this particularly for clients that are choking on server pricing--but you might also pay careful attention, because a robust Mono project will encourage/force Microsoft to compete on features and functionality, instead of a take-what-we-give-you mentality. That's a Very Good Thing.
Mono [go-mono.com]--an Open Source implementation of the
Re:Why are they running Windows then? (Score:3, Informative)
Yes, the Windows GUIs are significantly better than the UNIX GUIs. No, I don't think that matters. I've found that the Windows admins need to be every bit as clueful as the UNIX admins. Those GUIs hide the details, but they don't hide the concepts, and the concepts are still hard. There's no value in somebody clicking on the GUI buttons these days; modern systems are far too complex to get right by chance. The installer you mention is only 5% of the job.
At the end of the day your Windows admins are going to cost just as much as your UNIX admins. Quickly looking through the newspaper proves that point neatly; the salaries are within 10% of each other.
Dunno about .NET (Score:2, Informative)
Hardware: $10,000 USD (4x Dell servers)
Software: $0 USD
Bandwidth: $6,000/mo USD
Uptime >99.99%
64% CPU use on the single most loaded box
Sure,
~a
Re:LAMP is your solution. (Score:1, Informative)
So you basicly advocated redo ALL your software dev that you have been working on for the past year. Throw it out and just do it again. In the world of get it done yesterday. He would soon be out of a job. He will have to tweak till it scales.
He is finding out that optimization is HARD. Throwing more hardware at it has been a bad answer for years.
It sounds like he has some fairly decent hardware. He just needs to make sure that the vendor who sold it to him doesnt have some stupid junk running in the background. I have seen servers come from Dell and Compaq that have out of the box no less than 40 applications running. Thats the FIRST thing to do. Also be afraid of the 'RAID' answer. It buys you some but SQL Server has to be tuned to use it properly. Also more drives can help quite a bit with SQL Server. Just moving transaction logs, windows, the data, and the page file, all to seperate drives/arrays can give you a decent performance boost.
His next problem is actually sitting down and finding the bottlenecks. This is time consuming and takes lots of analysis. Timings and the like. Putting in places what call is doing what. He needs log files to help him. That sort of thing. Your next best tool for this is perfmon.
Another thing he can do if he does not like MSMQ. He probably would be better off making one. Its not that hard. A window with some threads hanging off it to pick up the workloads and a few postthread messages. He probably can get away with almost no locking that way. I have built MANY screaming apps that do just that.
Also make sure your not using the default heap managers in visual studio. They are rather piggish with multi threaded apps.
As for SQL server I have seen AWSOME impilmentations and HORRIBLE ones. Keep in mind ACID when making your tables. ACID [service-architecture.com]
Break the db into smaller tables. Look at the WAY you create datarows. How are you updating them. What sort of stored procedures are you using. Large table joins can help. But they can also hinder. USE the query plans to tell you HOW SQL server is trying to serve your requests.
Do you have to scan a whole table just to find a row? Maybe that would be better as some sort of a lookup thing that is loaded once.
Do you have tables that are 'key' tables. You probably do. These are the ones that will kill your performance if you get a few semi slow queries. You can see the locking behavior of this sort of thing. What if they were 'smaller'? Do you really need all the data in the WHOLE row all the time? Or is most of your time spent in small parts of the data. Then you branch out to the bigger pieces?
Remove things like 'binary' data out of your tables. You are using a database why put binary data in it? How will that help you get at that data if you have to have some sort of converter just to look at it? SQL server is very slow with this sort of data. Up to 20x slower just to read/write to those tables. You may find new uses for your data if you put it in a table instead of a single column.
2k
IIS
SQL Server
VB/VC/.NET/COM/WIN32/ASP
That is a solution also. It is a poor craftsman that blames his tools...
Scalablity is designed into an application. Its usually is not a factor of the OS.
For all you linux folk who are getting ready to flame me. Remember HE asked about MS stuff. Also MS DOES make some fairly cool stuff if you would stop and take a look at it.
Re:Christ, those machine figures! (Score:3, Informative)
Linus has a policy of not allowing multi CPU improvements to lower performance on single CPU. It doesn't get the cool press releases but those of us without million dollar IT budgets thank him for it.
Re:Ackward Worries, Threading and Responsiveness (Score:5, Informative)
A lot of folks are lambasting this guy because he wants to do C# inside SQL Server. Most are saying, like you, that he just doesn't get it because you should separate the database from the application. That's true but it doesn't invalidate the need to have stored procedures (in any language you want be it PL/SQL or C#).
The idea behind a stored procedure is that your application may actually scale better by putting some of the logic "close to the data" because there is less contention for machine resources other than CPU.
For scalability, it's *generally* true that you want no processing to happen on the database because database servers are generally more expensive to scale. However, moving selected bits of logic to the database tier can result in huge scalability improvements.
It's not one-size-fits-all and unless you have a good working understanding of the problem, which is impossible with the data given, it's probably not a good idea to yell "WHY THE HELL WOULD YOU WANT TO DO THAT" at someone. Give the guy the benefit of the doubt.
If you think that stored procedure in C# (or any
Yes (Score:5, Informative)
What im saying here, is that you are not the first person to ever consider how
Re: SQL server 2000
SQL server 2000 has more performance then you know what to do with, even on non-ridiculous hardware. Give it processors with lots of L2 cache (xeons) and lots of ram, and read all the docs about keeping MDF and LDF files on separate volumes (as well as tempdb) and you'll find that life is thrilling.
Data point: On a quad HT P4 Xeon with 8GB of ram and 12 spindles (a significantly less than $50k box) we support 1800 simultaneous connections, doing OLTP work against a ~15GB database. The most commonly hit table in the system has about 10 million rows that get added and deleted in batches of between 20 and 10,000, and updated singly or in bulk. Other apps select from this table on a polling basis (i.e. decision monitors). We could make our db and app design much "better" w.r.t performance, but we don't need to - the money we save not having to do genius level feats of programming, app rewrites, and perf tuning more than pays for the occasional new hardware or upgrade.
Continuing, Run perf monitor on your SQL server machine. Look at the physical spindle(s) that hold your MDF. If you're reading from them, buy more ram until you're not
You can tune SQL server without application changes until you're blue in the face, honestly. Use profiler to see what kind of queries you're doing. Put those queries in Query Analyzer and show the execution plan. QA breaks it down for you and shows execution time percentages of each sub-tree of the execution plan. If you've got something eating 80% of your time and its doing a table scan, do whatever you can to put some selectivity in that query (i.e. an index, or maybe a query change).
If you want to save yourself some headaches, setup management tasks to recalc indexes over the weekend (or nightly, if you see that much index fragmentation after a day).
Re:MS SQL replication (Score:3, Informative)
Some practical suggestions (Score:1, Informative)
ignorance made me post that incomplete post above (Score:3, Informative)
[shell]# mysqldump databasename > filename.sql
http://mysql.new21.com/doc/en/mysqldump.html [new21.com]
It can be done, with some hard work... (Score:2, Informative)
In order to do this you WILL (here's where I get flamed, but frankly, I don't care) need to examine and deal with the following:
1. How database intensive is your application? Despite claims by the DB suppliers RDBMS packages are slow.
Every query, even trivial ones like:
SELECT 0 as foo from bar
will require milliseconds to execute AT BEST. This is because you need to serialize the SQL request from your "middle-ware" (in this case the ASP.NET runtime instance) place this request into a inter-process communication channel (say a network buffer) then block. The network stack has to sent it over the wire, a thread on the SQL server machine has to be woken up to process the incoming query. The query has to be parsed, then executed. The results must follow the reverse route eventually waking up the blocked thread on the "client" (the machine running the ASP.NET runtime that initiated the call in the first place) parse the result set and present that result set to the rest of the application (via a "reader" object). At best this process is on the order of milliseconds per request. If you can get away with cleverly caching data in the application server then this task that originally took milliseconds now gets done in microseconds. Bottom line: eliminate calls to the database where ever you can. There is a down-side to this: more build time costs more money. Plus your code gets more complex. Evaluate the trade-offs. IF caching pays-off in your application it's a lot cheaper to add more application server boxes than it is to pay the crazy expensive prices to beef-up SQL server.
2. Multiple network cards per machine. When you did your benchmarking how much resource usage was there? On the machine running the Web application when you reached max transaction rate was the CPU at 100% If not you may be encountering either a network or DB bottleneck. Look at the DB was it maxing out (CPU + DISK), if so you need to focus your attention there, i.e.
3. Avoid using Web services "inside" your application. You may be using Web services to retrieve data from within your application code. Web services have a lot of friggin' overhead and are not particularly speedy. If you have to use Web services, then also try caching if you can. Any time you application has to wait for some other process (perhaps on a different machine) you will be incurring a big time penalty.
4. Where possible try to static-ify you output. This really is just another kind of caching. Many Web sites (including Slashdot) use this technique. It's particularly useful for news type sites. Write the output of your processing to the hard disk the first time it's requested then re-use this pre-generated file for subsequent requests. Update on a time-basis.
5. Yes, I'm going to say it: look at your DB schema and see if you have opportunities to de-normalize data to simplify queries. Joins are expensive, and query optimizers often don't do a great job. This approach can't always be used, but it can improve query performance orders of magnitude. What you may want to do is keep a really nice normali
I can't address all your questions, but ... (Score:2, Informative)
1. currently there isn't a mature messaging server and MSMQ is not appropriate for high load messaging platform.
No experience with MSMQ. I've used commercial pub/sub solutions to push over 3M messages a day to a 2 CPU Sun box.
2. SOAP is too damn heavy weight to scale well beyond 60 concurrent requests for a single CPU 3ghz system.
Maybe in the MS world. In the Sun world, we handle 400 concurrent over Gig Fibre. Kernel param tweaks were definitely needed tho.
3. SQL Server doesn't support C# triggers or a way to embed C# applications within the database
Use a real DB server. DB2, Oracle.
4. The through put of SQL Server is still around 200 concurrent requests for a single or dual CPU box. I've read the posts about Transaction Processing Council, but get real, who can afford to spend 6 million on a 64 CPU box?
A 64 CPU box isn't $6M. More like $2.8. Get a better supplier and if you are spending that kind of money, don't ever buy 1. You need at least 2. How else are you going to test and have a DR site? Don't put them in the same state either.
5. the clients we target are small-ish, so they can't spend more than 30-50K on a server. so where does that leave you in terms of scalability
Earning your consulting paycheck. I've deployed 450MHz 2 CPU Sun boxes that support 400 concurrent users (WLS), with a pub/sub incoming queue from multiple MF systems and retrieve on average 30 items from 8+ backend systems per user request. All while running a10 GB Oracle DB managing the incoming queues. To verify the scalability, we performed automated load testing on our test server. The production box is average 40% CPU loaded with peaks in the mid-70s%. It has been too successful, so the customer wants to add more functionality. This means we're splitting the app from the DB and adding a current generation 280R for each app instance.(3 boxes: dev, test/DR, prod).
6. I've been been running benchmarks with dynamic code that does quite a bit of reflection and the performance doesn't impress me. You've answered your own question.
7. I've also compared the performance of a static ASP/HTML page to webservice page and the throughput goes from 150-200 to about 10-20 on a 2.4-2.6Ghz system
You've answered your own question.
8. to get good through put with SQL Server you have to use async calls, but what if you have to do sync calls? From what I've seen the performance isn't great (it's ok) and I don't like the idea of setting up partitions. Sure, you can put mirrored raid on all the DB servers, but that doesn't help me if a partition goes down and the data is no longer available.
No experience. Generally, partitioning of time-based tables is a good thing when you are worried about data fragmentation. Better to drop an old table parition than have the entire table become fragmented. I've used a monthly partitioning successfully.
9. I asked a MS SQL Server DBA about real-time replication across multiple servers and his remark was "it doesn't work, don't use it."
There are other solutions, but they tend to be expensive. GoldenGate have an impressive replication tool - I don't know if it works with MS-SQL tho. It does work with all the big guys - Oracle, DB2, Teradata, so I wouldn't be surprised.
Re:Why are they running Windows then? (Score:3, Informative)
having said that, i think you are way off base. I used mySQL a lot and use SQL server extensively now. Comparing the two just isn't worth while. mySQL is a lot closer to berkeley DB than it is to SQL server, feature wise, scalability wise, management wise, and reliability wise.
SQL server has several licsensing options - per conection or per processor are the two default ones. Given that SQL server is perf competitive with Oracle, (it infact beats it on official benchmarks) and a HECK of a lot easier to deal with, SQL server is a STEAL compared to oracle pricing and support.
for most OLTP type work mySQL will NOT be faster than SQL server, especially as the volume of data grows, the complexity of queries grows, and the contention increases.
Put another way - when i see mySQL displacing SQL Server in the tpc benchmarks, i'll pay attention to you again. mySQL is very cool, but its a toy compared to DB2, Oracle, or SQL server. If all you need is a toy, by all means, use a toy -- you'll be happier. When you're ready to graduate to real systems, SQL server will be waiting for you. Maybe mySQL or Postgres will evolve faster than your needs do, and you'll never need a commercial quality database (or one of those two will turn into one)
It is unfortuneate that your experience has shown that Java+Jboss+{some db here, as you named like 5 of them}+Linux is "more reliable" than Windows +
Re:Why are they running Windows then? (Score:3, Informative)
I think I know what you are talking about.
I'm guessing this software is mainly decade old desktop packages that were originally designed to run on Paradox/dBase/FoxPro and ported to SQL Server or Oracle because that's the trendy thing to do. (If you see "BDE", the Borland Data Engine, it's a good sign that this is what you've got.) The thing is, the apps aren't really ported to use a RDBMS design. They still use "Flat Files" and have their own key/indexing system and old style coding.
The one I'm familar with is the very popular "Goldmine" sales package (had to get data from it's schema for an app I built). Doesn't even use Primary Keys, much less non-clustered indexes. Instead it's got these dbase-style bogo keys which look like "AAAA", "AAAa", "AAaa" and so on. But SQL Server is running in case insensitive mode, so all of the key comparison is done on the client! It also appears to do record locking on the client-side. No wonder an almost trivial application only supports 20-some users on a P4 server.
Hopefully as someone targetting DB2, you aren't making the same kinds of error, because you'll see the same issues no matter the RDBMS.
There are intentional problems between NT4 and Win2K+ using NT4 as a file server with SMB
I'm guessing this is the "rogue master browser" problem. Sucks, but an unofficially well known issue.
Oh yea, it can scale... (Score:2, Informative)
There have been a couple of posts w.r.t. proper architecture... While arch. is no doubt important, I've found that it is not hard at all to get good perf. out of even a minimal expenditure on arch. for
Here's where I've found the bottlenecks and easy ways to solve them:
1) Don't put SOAP calls in tight loops . If you seem to have to do this, redesign things. This may seem obvious, but the reason SOAP doesn't really scale well on any platform is because of the function call overhead, so minimize that as much as possible. Make sure to cache results local to the caller for stuff that doesn't change very often (see #4 below).
2) For DB updates, use SqlConnection transaction handling instead of MTS whenever possible. In the old ASP/COM+ world, it was imperitive to use MTS so you could take advantage of the other COM+ features, but the
3) Make sure to use the correct transaction isolation level for your transactions. For highly transactional apps. that also do lookups on the updated tables, the one I've had the best luck with is IsolationLevel.RepeatableRead. I can't stress the importance of this enough - do some rudimentary testing (Microsoft ACT that comes with the
4) Use HttpContext.Current.Cache and related in your code to cache stuff that doesn't change too often (like product description lookups, etc.), especially for SOAP calls.
5) Adjust the "Minimum query plan threshold for considering queries for parallel execution (cost estimate)" once you move your app. to an SMP system, depending on if the DB server is standalone and how highly normalized the database is. For a highly normalized DB (lots of JOINs) on a standalone SQL Server box, set this at 0 or 1 because most of your queries could probably benefit from a parallel execution plan, or at least the small extra overhead won't really hurt. This setting alone dramatically and immediately boosted the performance of one application and took all of 2 minutes to implement.
6) Make sure to apply indexes where it makes sense. For example, on one app. 4 hours of query research and index tunning based on that provided a 10 fold better response time for one query, which probably increased scaleability for that database by 100 fold because of how often that query was used.
7) Read and follow this simple advice for SQL Server clustered indexes on MSDN:
Have you tuned .NET and W2K? (Score:1, Informative)
I don't know whether anyone has posted this sort of info, and sorry, just didn't have the patience to wade through all the holier than though posts.
I recently finished up working with a government department which was deploying a number of
1. We often encountered issues with threading under
2. MSMQ seems to behave badly under high load. I don't know what to suggest in its place, but it ain't a great story.
3. With tuning we did find we could get pretty good CPU utilisation and prtty good throughput out of our front end
4. The behaviour of our system relied on back end systems. Under test conditions, when the back end systems were stubbed, throughput was good. However, under real conditions, were back end performance was abysmal (think MSMQ being used for calls to Mainframe etc) we found that behavrour could be unreliable - read long blocking calls seem to be a bit of a problem.
5. Other things we tunned included IIS (that stupid slider thing tends to give you 10% if moved to maximum) as well as the new connections buffer (all this stuff is in KB somewhere, and I guess you have done it).
6. Another seemingly not well known trick is increasing the number of user TCP ports. For some reason the default with W2K is only up to approximately port number 5000 is available for outbound connections (ie, you got some sort of soap middleware server listening on port 80, connections to it are 4998->80,4999->80,5000->80,OMG, I'm a teapot. You can set this up to the maximum allowed ~65000. Also changing TCPTimedWaitDelay will mean that used number can become available faster. Once again all this stuff is in KB.
Now, all that being said, I'm afraid I think your clients have made the wrong choice getting the big box. For frontend boxes the way to go is mulitple frontend servers for sure. I personally don't think the MS single process multiple thread model is all that good, although it can be made to work at least reasonably hard. Oh, BTW, you'll need an MS license for each of those front end boxes