Website Load Testing Tools?

bLanark asks: "I'm in the process of converting one of my clients from IIS/CGI to Apache/mod_perl. I need a (free) web site stress/load test tool to prove that performance will be increased. Using this page as my starting point, I can see that there are quite a few tools to investigate. Has anyone used any of these (or any others) and what are they like. I need HTTP GETS, form POSTing, and clever stuff like simulation of caching of images would be useful too, I guess." The previous story didn't get much of a response, but that was about a year ago, but the submittor has shared a fairly impressive list with us, impressions about any piece of software on it would be greatly appreciated.

Easy

[epsalon]

Post it on the front page of slashdot...

New unit of site capacity

[GCP]

"By upgrading from IIS to Apache/mod_perl, we were able to increase our load capacity from 300 millislashdots to a full 1.2 slashdots while cutting costs by nearly two-thirds...."

Re:New unit of site capacity

[dacarr]

But what would constitute a slashdot?

Re:New unit of site capacity

[green pizza]

But what would constitute a slashdot?

Watch for the browser distribution to shoot from 0.01% Mozilla to about 95% Mozilla!

But seriously... a Slashdotting usually takes the server from light load to an extreme load in a matter of seconds... remaining at max load for about 4 hours (~2 million unique visitors an hour) and then dropping back to normal over the next 12 hours.

Re:New unit of site capacity

[dacarr]

This is true, but then that's the result of the act of slashdotting. The machine is brought to its knees, and slowly recovers. What I'm wondering though, is basically what would the unit "slashdot" be comprised of - perhaps base it on an average concentration of user per time unit? "Woah, flash crowd!" "Don't worry, it's just a few millislashdots, we can weather this."

Re:New unit of site capacity

[Kredal]

Let's see. 250,000 people come to Slashdot every day. We'll assume you get a story posted in the morning, right before everyone (well, nearly) gets to work, and they all log onto their favorite site (nonono, not pr0ns-r-us.com), and 1/3 of the users click on the link within an hour of it being up.

That's 83,000(rounded) views, if you have a single page with no pictures. I suppose we can use that as a basis, and people can calculate out from there, if they want to. So:

1 Slashdot (Sd) = 83,000 views (per hour).
1 CentiSlash (cSd)= 830 views
1 MilliSlash (mSd)= 83 views
1 MegaSlash (MSd)= 83 million views

The average high volume news site (CNN, ABC, Register) can probably withstand 2-3 Sds, while a potato powered, or C64 server could only withstand 10-20 cSds. Of course, the lower powered servers will have less to download, so a single Sd may be managable.

Again, this gets into how many pages/images Slashdot is linked to.

Further study is advised.

Re:New unit of site capacity

[SEWilco]

A slashdot is a whole unit and always 1. Whatever activity you get on your site after being posted to the front page is 1 slashdot.

This includes the activity which gets bopped around the Internet and doesn't get to your server.

Re:New unit of site capacity

[dacarr]

Hmm...

Perhaps this should be adjusted, IMHO. 1 slashdot could be the number of simultaneous HTTP requests that is needed to take down a machine, with a varying scale. This is variable though - some machines (remember our gamer table friend) can't withstand much activity before melting, others can take insane loads, run SETI, run a Quake server, recompile their code, process multiple MySQL requests, and execute endless loops in less than one second without so much as blinking. (Hey, I want that last box!)

Great, thanks to slashdot, I'm on the search for a defineable unit =/

Designing the tool

[DeadSea]

I don't know what is out there, but if I were picking a tool, I would make sure that it measures real world cases. This means that you should be able to give it your server log for a week (make sure you log post data and cookies) and the tool would test your server just like it is actually used. It would also be able to test for increased traffic by combining several weeks at a time (Like all the visitors that came on 4 Wednesdays came on one day).

I would also think that it would have to be distributed (at least across several computers) to ensure that bandwidth and cpu/memory of the testing computer are not the limiting factors.

How hard can it be to write something like that? The hardest part would probably be parsing the log files. I would not be suprised if there are several testing tools that would do an adequate job.

wast

[Anonymous Coward]

For the simpel testing that i've done, i've always used http://webtool.rte.microsoft.com/ . It's free, easy, quick (nothing like 100 hit/s to bring a site down) and does what you want. It has lots of fancy features only a point and a click away (yes, it can be scripted to), but I've newer had the need for it. I only use the stress tool the see *if* there is a problem, you then need a profiler to see where the problem is.

Apache comes with one

[Quixote]

Checkout Apache Test [apache.org] page. Apache comes with Flood [apache.org], a profile-driven web server tester.

Be sure to simulate "real life"

[linuxwrangler]

At least as much as possible.

In a previous job we needed to do some benchmarking and testing. The first pass was to script some stuff using wget. Once everything looked fine (we could really slam the server) we turned it loose on the public.

Oops. The site collapsed nearly instantly. The problem was that people with slow modem connections kept connections active for a couple orders of magnitude longer than happened on our internal network and the server ran out of resources.

Microsoft's free tool can simulate a mix of connection speeds and I believe you can find similar functionality in many of the web test tools you will find in a freshmeat.net search.

MS's free tool. You mean Homer?

[barzok]

We tried Homer. It was OK, but we very quickly came to the ends of its capabilities. It doesn't simulate an actual user; rather, it acts as a proxy when you record your script, watching all your HTTP requests. It then plays that back, and you can parameterize some of the values.

What it doesn't let you do is make a hop from HTTP to HTTPS, nor does it simulate caching of images, nor many other things that an actual user would do. JavaScript is right out. For small, simple apps, it worked OK, but start getting complex and Home just can't handle it. D'OH!

It isn't free, but...

[abulafia]

OpenDemand [opendemand.com] has a fairly sophisticated test suite packaged as an ASP style offering.

-j

Simulating "real life" is tough.

[Anonymous Coward]

The problems are never what you expect. After hypertuning the web server you may find that your SQL backend is causing a bottleneck. Or that something in an external, offline, transaction- finishing process is flaky/unreliable/slow. Or that one particular area used by one particular class of users is making their perceived response horribly slow, while the rest of the world notices no problems at all. One thing that has helped on an aging site I work with is to track percentage of "completed" actions - for example, the user should click pages 1/2/3/4/5, in that order. If 80% make it to #3 and 15% are getting to #4, that may reveal bottlenecks that ps and ab and top won't show you.

Re:Simulating "real life" is tough.

[Anonymous Coward]

"... the user should click pages 1/2/3/4/5, in that order. If 80% make it to #3 and 15% are getting to #4, that may reveal..." that the information you are presenting on page 3 is helping them decide that you don't have the product which they want (if you're selling somethign). As a customer, I like having enough information for a decision so I won't waste my time. As the webmaster, you might want to provide a "Not what I want" button to try to collect feedback about what is happening. That button would at least tell you that those who click it are leaving on purpose. You could provide several such buttons for various possible decisions, some of which might lead to related categories in your catalog, or might just go to a page with "Thank You for considering us...here is a link to our Home Page if you want it, or please select one of the following buttons to help us serve you better in the future (followed by several possible reasons for their decision to leave the site)".

Oh -- another way to detect bottlenecks that 'ps' won't show you is to wrap all your CGI with start/stop log commands. When you turn on a setting it causes all your scripts to log start time, end time, CPU time, etc. Helps you find where to study further.

Siege

[modecx]

I have had good luck with the neat little program called siege [freshmeat.net]. It can stress a single URL, multiple URLs, follow links from a root URL (simulating an actual user), and have many multiple concurrent connections active. At the end, Siege can tell you all about the server performance, latency, etc.

I really like one of the other poster's idea about having a load tester read actual log files from Apache, then simulate real user activity. The only problem I can see with this method, is if you changed the layout of your site, all the program would get is a bunch of 404s. However, if one were so motivated, one could hack up such a thing relatively easy, I think. analog [analog.cx] can parse Apache/httpd log files, could'nt be all that hard. Siege works well for me, though, so I'll stick with it.

Re:Siege

[Foozy]

I second the use of siege. I've found it to be easy to use, very fast, and customizable. There is another piece of siege called 'scout' which is used to scout out a website and return page entries for later load testing by siege. Scout is also customizable.

Using scout and siege, with even a simple shell script, you can blast the bejezus out of a web site. (Oops, may not want to say that on slashdot...)

Re:Siege

[modecx]

It's been a while since Iv'e needed to test a site, but I gotta say that siege has progressed since I last used it. Thanks for the tip on scout, looks like it makes life even a little bit easier :)

JMeter is cool...

[PinglePongle]

though it seems to run out of resources at a few hundred concurrent users. You can find it at jakarta.apache.org. It's not very sophisticated, but it can certainly perform basic load testing.

I've also used (Mercury's) LoadRunner, a fairly expensive and complex beast; it comes with a built in proxy which allows you to capture a browsing session as the basis for a test scenario - this is a particularly useful feature. The scenarios can be scripted, and there's a bunch of random timers you can use to represent "real-world" behaviour.

I'd suggest creating 2 types of test scenarios : 1 type to represent your best guess at "real world" usage, i.e. representing users browsing, spending time reading a page, clicking all over the shop etc. - this should give you a good idea of how your application is going to perform.
The second type of test should provide you with metrics you can use when making improvements once you spot bottlenecks. They don't necessarily represent "real-world" activity, but allow you to measure whether code changes have made a difference. For example, you might have a test where all users log into the application simultaneously. In version 0.1 you can support 10 concurrent users; in version 0.5 100 concurrent users, and in version 1.0 250 users. These numbers don't represent "real" users - how likely are they to log in all at the same time ? It's a lot easier to manage your performance improvement work using these metrics.

Re:JMeter is cool...

[DigitalRonin]

I've used Jmeter - great tool.

FYI, it also does the http proxy recording thing now.

Check out The Grinder

[Solaristrum]

The Grinder (http://grinder.sourceforge.net [sourceforge.net]) is an open source Java based load generating tool. It is multi-threaded, distributed, scriptable and has a central statistics reporting mechanism (console). It was used extensively by the authors of J2EE Performance Testing with BEA WebLogic Server and that book has excellent instructions and recommendations about its use.

I suggest you check it out.

JMeter, Siege, Mercury Interactive

[bcarlson]

I've used both JMeter and Siege and liked the results (fairly accurate, easy to use, scriptable). I've discussed some of the high cost solutions such as offerings from Mercury Interactive[mecuryinteractive.com] [mercuryinteractive.com] with developers I trust, and the concensus seems to be that if you use several of the testing tools avialable, grab some humans at the same time to check the URL, and take all of the results with a grain of salt, you'll be better off than spending beaucoup bucks on Mercury Interactive, or similar tools. (Note: I have never worked for Mercury Interactive, or used their tools) Also, I have tried the Micro$oft tool, I thought it sucked, but I'm an open source biased individual... :)

Good luck!

The Grinder, again

[AdamInParadise]

Like a previous poster, I recommend The Grinder. I looked all around for a load testing tool. People usually recommend the Microsoft tool, but I don't do Microsoft. I tried JMeter and a bunch of other tools, but they never feel quite right. Most tools fall in two categories: either completly unscriptable or so scriptable that using a general-purpose script language would be faster.

Among (many) other things, The Grinder has a built-in proxy that allows it to record a browsing session and play it back later. Basically, you start the proxy, set your browser to use this proxy, browse your website, and get back a log of your actions complete with timings and POST values. One other cool features is that it let you define your own datasources so you can fill POSTs and GETs with custom data. Last but not least the author of the tool will personally answer your questions, albeit slowly.

Burn Karma Burn...

[TheTomcat]

[joke]

Check out the WAST. It's free, and IMHO, better, and easier to use than ab (apache benchmark)... although, it's comparing tangerines to oranges.

http://webtool.rte.microsoft.com/ [microsoft.com]

S

The tool to use

[smaughster]

If you want your tool to be free as in beer, then I can't help you, but if you want to know what one of the best tools for the job is: mercury interactive's loadrunner. I work at a test department which has a sub department specialized solely in load and stress tests (and our testresults often show painfully enough that loadtests experts are very much needed) and they wont work with anything else. It is easy to build a small tool that will generate load, but if you also want to easy analyse the results, reuse test cases, integrate your graphs with other measurements (webserver memory, cpu, I/O etc)and want to breakdown the results in server time, network time etc then you'd be very happy to use a professional tool. Only caveat: it's expensive.