Web Performance and QA Tools?

perf_monkey asks: "I'm part of a large Web Infrastructure Quality Assurance (QA) team at a large financial institution. Currently, we use Mercury Interactive's LoadRunner like a bunch of trained monkeys. We also use QA Load and SilkPerformer, but for smaller (non-J2EE) projects. As one of the technical folks, I've been trying to expand our horizons and our budget. I can no longer believe that large companies are willing to pay a QUARTER OF A MILLION dollars for the privilege of an additional 2000 Mercury VUsers. I'm looking at both commercial and open source alternatives. I've been tinkering with The Grinder and have had pleasing results. While not a full-blown QA tool, it is an excellent 'programmer's' load test tool. I was hoping that there are other tools like this and was hoping for the community's opinion. What web performance tools do you use and what do you think of them?"

Under windows?

[trompete]

I'm looking for the same kind of tool but for Windows applications. Any guidance is appreciated.

Re:Under windows?

[bobbozzo]

Most perl-based stuff will work under Windows with ActiveState's ActivePerl (free as in beer).

Question

[Gildenstern]

Well I have to say I don't know anything about this type of application but typically what type of requirements do you have for a system like this.

How does lode runner test systems now?

I work

[CptChipJew]

for a major software company, and we have licenses for Mercury WinRunner, LoadRunner, QuickTest, Rational TestSuite, and even a product we wrote ourselves. This comes out to millions of dollars in software and maintanence licenses.

Now, I'm not saying the spending isn't wasteful. I think it is, but it's not correct to say that large companies are paying for expensive SQA software licenses anymore.

Perl and Java

[jo42]

Perl and Java are your friends...

Web Performance Trainer vs. OpenSTA vs. Mercury

[Fudge.Org]

Disclaimer: The company I work for is a very happy WebPerformance Inc. customer.

I've used Mercury products for unit testing, full out scaling tests, monitoring, and defect tracking for over three years. The problem was the total cost of a Mercury load test bed. It gets expensive rather quickly unless you have planned in advance or do a lot of Mercury work. If you don't it makes more sense to hire a consultant that has an open license arrangement. That way you only pay for when you use it. YMMV.

So, I looked into some other alternatives and ended up going to unit test applications from the Apache group and home grown benchmarks to approximate load scenarios.

Eventually, I ended up looking into OpenSTA OpenSTA [opensta.org] and was fortunate enough to work with some real experts that did a lot of load testing work with OpenSTA. There are some limitations to the OpenSTA *cough* Windows only *cough* but overall it is an excellent tool for driving load for a complex web application. Much like Mercury or any load testing tool the test is only as good as the planning and analysis you perform to isolate performance issues.

My most recent job called for load testing but in that time Mercury changed their license program again and we really wanted to use something that would run on Linux. So, OpenSTA was not really in the running. Also, we needed to simulate multiple IP addresses not just hundreds of virtual users from a single source address. (fine for simulations of a access from a corporate firewall). So, I had to find something that would work and that wouldn't blow our budget.

I checked back with a company called WebPerformance Inc [webperformanceinc.com]. Now, I looked into a few years ago to see if they supported IP spoofing for virtual users and SSL. They had done SSL but the IP spoofing wasn't done.

As it turns out, they put this feature into their 2.6 release. We use it to run our large tests for burn in and acceptance for revisions to our network hardware that provides web interface. Like Mercury, you can really ramp up serious traffic. Couple this with basic network load generation and you can create a sound simulation of network throughput and application access from multiple network addresses with mulitple users and multiple business cases. Oh, and the fact that you can get a price list that is straight forward is very nice.

So, I'd say each is right for a certain type of test and a certain type of shop. For our shop, Mercury is cost prohibitive, OpenSTA lacks a key feature (IP Spoofing), and WebPerformance, Inc -- while commercial -- satifies our feature requirements near perfectly.

Open STA?

[Kanagawa]

Have you looked at OpenSTA [opensta.org]? Our development team in NH uses it and they enjoy it quite well.

Re:Open STA?

[Fudge.Org]

OpenSTA is great (and free!) software. The fact that it is on SourceForge is an added bonus.

Windows only

[michaelggreer]

The fact that it is windows only is not great. Otherwise, I agree. It is the tool I use in windows environments.

Re:Windows only

[Fudge.Org]

Yes. Still, it is possible to run inside VMware instances (very much not ideal) if you need to encapsulate or stagger testing servers across a small subnet to ensure portability of a test scenario across different source IP access for the virtual user block.

It's not impossible just slightly cumbersome. :)

Web Performance Inc.

[raimundo]

I generally don't appreciate proprietary software, but I've actually liked Web Performance Inc's tool.

http://www.webperformanceinc.com/

They've been quick to respond to our concerns, and their price was significantly cheaper than any of their competitors.

Load Generators

[xanthan]

While hardly comprehensive test tools, there are some nice load generation tools that have some pretty remarkably features for their price (free). Check out WebBench (www.webbench.com) which does a nice job of harnessing a ton of machines to generate as much load as you can dream up. Then there is WAST (Web Application Stress Tool) from Microsoft. WAST has some nice features as far as scriptability goes, you just need to be warm and fuzzy with WSH to do it. On the Unix side you can use things like http_load

Save you money

[Books]

I was a VP of IT for a startup, developing a large content web site. Initially the site was hosted on 2 Pentium based IBM servers, running Linux. The server were no where near full capacity, but we wanted to check how much load these machines could take before requiring additional servers.

Solution such as LoadRunner were too expensive, but even renting LoadRunner users for a few days was too much. We figured that for the money we saved by not checking the load we could buy and host two additional servers f

use curl

[oo_waratah]

I wrote a cywin script using curl to access the server. I signed on 100 users and tracked each response. I then did a few different paths down an application chain and then did a few major image loads. I could get response times for each screen and using 10 PC's around the place running 10 instances I had a simple load test. (25 bash scripts appeared to blow a P4 windows, but cruised on a PII350 pure Linux)

OK this is not comprehensive and slick but it worked sufficiently to prove we could handle 100 c

Re:use curl

[corporatemole]

Or, your better performance with 50 users could be attributed to errors in the application. I've seen web apps that when an error occurs, it doesn't return a 500 level error message to the browser. Sometime an application "handels" an error (which is caused by something like hitting it too hard and maybe it doesn't have any available database connections) and it returns back to the browser really quickly. If you're only checking HTTP status codes, you may not be seeing real errors.

Reduce the scale rather than increase the licenses

[PinglePongle]

I used to be the development manager for a content-based website, and we had significant performance issues.
We had a limited budget, and buying even tens of thousands of dollars worth of licenses was - in our view - a bad use of that budget. Instead, we concentrated on building a test infrastructure with known performance characteristics, and extrapolated (sorry, I meant "guessed") that this performance profile would scale by a factor of X on the live infrastructure. By using a relatively small and flexible

Uh.

[TheLink]

Just post us a link whenever you feel like a good slashdotting :).

Open Source Testing Tools

[corporatemole]

Here [opensourcetesting.org] is a website that maintains a list of open source testing tools. Some of the tools mentioned in some of the comments here are in that list.

Home-brew automation guru

[sohp]

Interesting timing on this question. Just recently I attended the Pacific Northwest Software Quality Conference [pnsqc.org] and heard a talk by Bret Pettichord [pettichord.com] on just this very subject. His presentation, Home Brew Test Automation [io.com] covered this subject with some terrific lessions.

Custom HttpUnit code

[dubl-u]

Here's a trick I stole from agile methods like Extreme Programming: build the test suite up at the same time you write the main code.

The main reason testing existing applications is so hard is that they aren't designed to be easily tested, so you have to jump through a lot of hoops to do proper testing. However, if the developers don't get to check anything off until their components are covered by unit tests and their features are covered by acceptance tests, that gives the developers the incentive to mak

Re:Custom HttpUnit code

[babbage]

A/k/a "requirements driven development", among other names. It's generally considered to be a sound development strategy for any kind of software development. First you figure out -- if only roughly -- what the software needs to do, then write tests that will verify that a given implementation will work, and only then do you start working on the actual system itself.

Once development is under way, the goal should always be to make it correct first -- and you know it's correct because it passes the tests (p

Easy!

[blackmonday]

Get your website into a story on Slashdot. A case beer and an XL Pizza should be enough bribing for a real geek. Get out your stopwatch, measure the seconds before you smell something burning.

Repeat and tweak as necessary.

THANKS for your help

[perf_monkey]

I just wanted to say thanks for everyone's suggestions. I consider myself a pretty experienced perf expert and I learned about some good OS tools. I'd like to give a bit more of a background of my setup.

My PTE (Performance Testing Environment) is made up of an AWEFUL lot of REALLY expensive Sun boxes. That is where I use Mercury, and will continue to use Mercury. I have another setup for the DIT, SIT, and DEV environments. The load generators that I use in the aforementioned environments are 5 beefy x86 R