Software QA and Load Testing Solutions?

tekiegreg asks: "I've been asked by the boss to evaluate Load Testing and QA solutions for use by our company. Google Searches have yielded TestComplete and Mercury's solutions. However prices are very steep. Has anything in the Open Source world even come close to this level of functionality in a testing suite? Searches of Sourceforge and Freshmeat reveal nil. Are there any other solutions that people have tried, out there?"

JRunner

The only open source solution I'm familiar with is JMeter. I haven't worked with it much, but in the few projects I've used it, it's been helpful. I't part of the Jakarta project. You can find it at http://jakarta.apache.org/jmeter/index.html [apache.org]

JMeter

JMeter [apache.org]

you need something like LoadRunner

The closest open source testing tool I have found, that resembles Load Runner the most is OpenSTA: http://opensta.org/ [opensta.org]

Mercury is worth every dollar

Of course, experiences vary, as do needs. Our experiences with Mercury's managed testing were worth every dollar. We did investigate open source options at the time, but we just didn't have the available developers to work out the test scripts.

Uhmm, perl?

I don't think you'll find many open source testing tools. Usually open source software is tested by its users. While that is hopelessly inadequate if you have 5 users, and intend to charge them big bucks, it tends to work well in the open source world.

Not open source....

Worksoft [worksoft.com] has a proprietary solution. I don't know if it's cheaper than TestComplete. I just happened to see it today.

Rational?

I use Rational's (IBM) stuff in my work. It's expensive for sites, but I think pretty reasonable if there is only going to be one tester.

Rational Robot does automated testing. I'm pretty sure they have load and performance tools too.

http://www-306.ibm.com/software/rational/ [ibm.com]

Do it yourself

I have 5 years of experience in this field.

Get it over with and just write the software yourself. Most of the work is custom software anyways.

Log everything to an xml file and or a database.

Write your own front end to the whole test harness.

There really isn't very much good help available.

Even though nunit is good for unit tests, there is nothing that stops you from implementing load/stress and performance tests with it.

You can even use the nunit user interface as a testharness until you get your own written.

In the end you'll spend some time and money on your own automation, but that is what you would have ended up doing down the road anyways.

Some Open Source Testing Tools

Watier [clabs.org]

The Grinder [sourceforge.net]

Selenium [thoughtworks.com]

Last January there was a workshop on open source web test tools [pettichord.com] in Austin.

Depends on Your Requirements

Load testing and QA have various meanings. If your QA simply means that you track the delta between load tests, and if the delta isn't significant a release is approved then something like OpenSTA should be just fine for you.

However if you need to be able to do regression testing, performance tuning, and code profiling then you will either need to drop some cash (Mercury can do all these things) or spend some considerable time doing your own development. My company developed something similar to Mercury's load runner and quick test, and looking back it would have probably been less expensive to just spend the money up front.

So, if you don't have money but your requirements aren't real strict then a hodge podge of open source tools with a bit of custom development will work fine. If you have strict requirements then be prepared to spend a bunch of time or money getting these tools up and running.

Have you considered...

Have you considered Linux and some Perl scripts? Even little olde wget can generate a considerable amount of traffic for you.

Emily Latilla might say

Since IBM is so pro-OSS they're releasing the source for Rational Visual Test a popular product they discont... What is that? You mean IBM is not releasing the code? But I thought you couldn't buy any more licenses for it? You mean IBM says OSS stands for Old Slippery Snake-oil?

That's very different. Never mind.

At my last programming position

My employer used an modified (in house) version of CCUnit [sourceforge.jp] to automate testing on Solaris. For Windows testing, we used WinRunner. But that was two years ago. CCUnit claims to be platform dependent, so it might work on Windows.

Write it yourself using Perl and IE

I've seen a lot of comments mentioning you should write a testing tool yourself. If you decide to do this, maybe this article [perl.com] could be useful; it talks about automating IE using Perl scripts.

JMeter

Have you looked at JMeter [apache.org]? I've used it to test web/server applications, but it can be used for other applications too.

Grinder

For web apps, we've had a great deal of success with grinder [sf.net]. Its scripts are easy to write, and its recording mode is sometimes all you need to gin up a quick load test. For one project, we had Mercury as well... we found that grinder was much more helpful, at least for the development team. (Though I appreciated having both.)