Stress and Volume Testing - Your Experiences?

Tarohann asks: "I work for a large software services company and I'm planning to write an article on Enterprise application Stress and Volume Testing (SVT) based on a couple of experiences I've had. For the most part I feel that most Enterprise SVT initiatives performed by companies at the enterprise level are flawed as they are done on ideal/non-production environments. I was wondering if anybody here has had any good/bad experiences or pointers (websites, books) I can refer to on this subject. Full credits will be given. Thanks, Slashdot!"

A precise question deserves a precise answer...

[stienman]

I was wondering if anybody here has had any good/bad experiences or pointers (websites, books) I can refer to on this subject.

Wonder no more. There are people here who have had some good/bad experiences or pointers that you can refer to on this subject.

-Adam

Re:A precise question deserves a precise answer...

[SpaceLifeForm]

For example, I've experienced major stress doing volume testing.

Volume testing

[A nonymous Coward]

Would that be pints or quarts or liters or fifths or ... ?

Want to get a real environment?

[dmayle]

Gather 20 of your closest friends and relatives, give them each a different OS cd, and a different used computer (some laptops, some desktops, different brands, etc.) and have them each install the OS to their liking. After that, let them swap and repeat. For better results, image those systems (with ghost, etc.), and install the images on other systems with different hardware, and do whatever it takes to get them going on the new system. Finally, take a couple of the systems, have someone who didn't work on it go over to it and remove the crap that the other person put on it, and install it to their taste.

After this, you will finally have an environment that reflects what really happens in most computer networks. Image again, and save one image per machine, and you should be all set. (Unless you want to swap some hardware around, like NICs, or modems)

I work for a company that writes systems management software for windows and *nix systems, shameless plug [criston.com], and each of the QA machines have a different OS, and a mix of languages for color. We've got solaris boxen, macs, linux in many different flavors, and every x86 Microsoft OS released since September '95 (except Windows ME, 'cause no one really counts that as an OS ;) )

We've got used computers from many different vendors, servers, desktops, etc., and let me tell you that we encounter many more issues in captivity because of it...

Sun knows enterprise SVT really, really well

[joelparker]

If you can afford it, get help from Sun.

The Sun application server software team
has people with very deep backgrounds
and the right experience and equipment.
I was truly impressed working with them.

They do massive testing projects
and have all the necessary contacts
with other enterprise app companies.

Cheers, Joel

Do my job for me

[Myco]

Can we do your kid's homework, too?

Re:Do my job for me

[arohann]

I didn't ask you to my job for me. All I asked for is if you've had any good/bad experiences or opinions with SVT and what they were. /. would be a better place if you'd just read the post and reply meaningfully and tried not to be a moron.

I have...

[lowmagnet]

The best thing I can say to you is that if you want to 'sell' the test results to your management, etc., you are going to have to do both database/application server capacity and load testing and user-side 'performance' testing if you have a user-side to your application. For some reason, PHBs on up like to see the little application animating its brains out. I'd suggest Mercury Interactive's WinRunner and LoadRunner, but I was too lazy to RTFQ and beside that, I've been coding in WinRunner all day, and I now need beer to soothe my nerves.

I like how P&CP (performance and capacity planning) becomes S&VT when you are talking "Enterprise" applications. It's funny how far people will go to sell a new approach, or to pad their resume with more buzzwords that nobody understands.

Stress and Volume - Your Experiences?

[the eric conspiracy]

My boss used to give me stress, and a very high volume of it. My experience was I didn't like it one bit, so I found another job.

experience

[pizza_milkshake]

i've had trouble testing my volume, and it's stressful.

Bleah.

[dbirchall]

Last time I was involved in that kind of testing, we found out that our code worked as designed (yay!). Unfortunately, we also found out that the application server engine we'd written it for had some critical bugs buried very deeply, that had never shown up because no one had ever tried to do anything quite that stressful with it. :( Oh well.

Ideal environments are your fried sometimes!

[basking2]

Sometimes ideal environments offer higher throughput and stress on a system. Consider a webapplication on a busy network with one interface. Very typical and much lower stress than you might expect. Consider the stress-volume environement: You could have 3 nicks on a quiet switched segemnt and HAMMER the server.

Obviously, you want to profile your testing environment while testing to find a bottle neck and work it into your analysis. During one testing experience we found that the IDE drive started working very very hard and limited our throughput!

Just some thoughts. Home you find it interesting.

Re:Ideal environments are your *friend* sometimes!

[basking2]

That will teach me to proof read my posts more closely.

Extrapolating worst-case loads

[G4from128k]

You might look into using something like the Pareto distribution [wikipedia.org] or Zipf's Law [wikipedia.org] for estimating the probability of low probability, high-magnitude loads on the system. A little math like this in conjunction with a limited sample size of observed input load levels will help you guesstimate the 99.9999...% level of loading.

Some sites to try

[nightgeometry]

I'd suggest asking your question at Sticky Minds [stickyminds.com] (forums require registration - they haven't spammed me yat after a lot of years), or QA Forums [qaforums.com].

Mercury Interactive

[Asmodeus]

I'd strongly recommend a section on this company. They have a number of load generation centres distributed around the world, and can hit your live site with a huge load (practically a denial of service)

Their engineers are very experienced with load testing and their product Load Runner is the defacto standard for stress testing.

I dont' work for them, but I have worked with them and been very impressed.

Asmo

Pointer to software...

[okock]

See OpenSTA [opensta.org], an Open Source load testing application for web applications, with some more links in the WebLink section on portal.opensta.org [opensta.org]