Software QA and Load Testing Solutions?

tekiegreg asks: "I've been asked by the boss to evaluate Load Testing and QA solutions for use by our company. Google Searches have yielded TestComplete and Mercury's solutions. However prices are very steep. Has anything in the Open Source world even come close to this level of functionality in a testing suite? Searches of Sourceforge and Freshmeat reveal nil. Are there any other solutions that people have tried, out there?"

JRunner

[Joe Helfrich]

The only open source solution I'm familiar with is JMeter. I haven't worked with it much, but in the few projects I've used it, it's been helpful. I't part of the Jakarta project. You can find it at http://jakarta.apache.org/jmeter/index.html [apache.org]

Re:JRunner

[Intron]

So what does the poster need that jmeter doesn't do? Here's the thing: ITS OPEN SOURCE. Maybe an additional needed capability could be added and returned to the OS community.

People (and I won't say just Windows users, mind you) have a tendency to look at OS software and say: This doesn't do what I need, so I'll just buy the commercial product. It might be cheaper to write the extensions to jmeter.

Re:JRunner

[Anonymous Coward]

This could be true, however

1) It could be possible that the organization might not have the resources to develop (not just money, but expertise) such extensions

2) The time to develop, test and deploy is probably going to be longer than a buy (or download) and implement. What is the time frame?

3) The cost to benefit might still be better in buying a commercial package

Re:JRunner

[dubl-u]

The cost to benefit might still be better in buying a commercial package

It might. But beware. If you don't have the source, then you'd better be doing something firmly within the envelope of what the designers expected.

JMeter

[NastyNate]

JMeter [apache.org]

you need something like LoadRunner

[mozkill]

The closest open source testing tool I have found, that resembles Load Runner the most is OpenSTA: http://opensta.org/ [opensta.org]

Mercury is worth every dollar

[Bravo_Two_Zero]

Of course, experiences vary, as do needs. Our experiences with Mercury's managed testing were worth every dollar. We did investigate open source options at the time, but we just didn't have the available developers to work out the test scripts.

Re:Mercury is worth every dollar

[commanderfoxtrot]

I use Mercury Loadrunner frequently, with thousands of virtual users.

It's a very powerful tool but the essence of it is very simple.

I looked at OpenSTA, but it is simply not in the same league.

The Mercury licences cost many tens of thousans of pounds, but do note you can buy short-term licences which can help reduce costs. When you're working with multi-million-pound projects, the costs are well worth it.

Uhmm, perl?

[joto]

I don't think you'll find many open source testing tools. Usually open source software is tested by its users. While that is hopelessly inadequate if you have 5 users, and intend to charge them big bucks, it tends to work well in the open source world.

Re:Uhmm, perl?

[kwoff]

Usually open source software is tested by its users. ... it tends to work well in the open source world.

For example, witness that paradigm of open-source code, slashcode, tested by hundreds of thousands of users every day. Quel joie!

Not open source....

[Nick of NSTime]

Worksoft [worksoft.com] has a proprietary solution. I don't know if it's cheaper than TestComplete. I just happened to see it today.

Rational?

[CptChipJew]

I use Rational's (IBM) stuff in my work. It's expensive for sites, but I think pretty reasonable if there is only going to be one tester.

Rational Robot does automated testing. I'm pretty sure they have load and performance tools too.

http://www-306.ibm.com/software/rational/ [ibm.com]

Do it yourself

[abradsn]

I have 5 years of experience in this field.

Get it over with and just write the software yourself. Most of the work is custom software anyways.

Log everything to an xml file and or a database.

Write your own front end to the whole test harness.

There really isn't very much good help available.

Even though nunit is good for unit tests, there is nothing that stops you from implementing load/stress and performance tests with it.

You can even use the nunit user interface as a testharness until you

Some Open Source Testing Tools

[pfdietz]

Watier [clabs.org]

The Grinder [sourceforge.net]

Selenium [thoughtworks.com]

Last January there was a workshop on open source web test tools [pettichord.com] in Austin.

Depends on Your Requirements

[matheny]

Load testing and QA have various meanings. If your QA simply means that you track the delta between load tests, and if the delta isn't significant a release is approved then something like OpenSTA should be just fine for you.

However if you need to be able to do regression testing, performance tuning, and code profiling then you will either need to drop some cash (Mercury can do all these things) or spend some considerable time doing your own development. My company developed something similar to Mercury's

Have you considered...

[jo42]

Have you considered Linux and some Perl scripts? Even little olde wget can generate a considerable amount of traffic for you.

Emily Latilla might say

[ClosedSource]

Since IBM is so pro-OSS they're releasing the source for Rational Visual Test a popular product they discont... What is that? You mean IBM is not releasing the code? But I thought you couldn't buy any more licenses for it? You mean IBM says OSS stands for Old Slippery Snake-oil?

That's very different. Never mind.

At my last programming position

[brokeninside]

My employer used an modified (in house) version of CCUnit [sourceforge.jp] to automate testing on Solaris. For Windows testing, we used WinRunner. But that was two years ago. CCUnit claims to be platform dependent, so it might work on Windows.

Write it yourself using Perl and IE

[cerberusss]

I've seen a lot of comments mentioning you should write a testing tool yourself. If you decide to do this, maybe this article [perl.com] could be useful; it talks about automating IE using Perl scripts.

JMeter

[unboring]

Have you looked at JMeter [apache.org]? I've used it to test web/server applications, but it can be used for other applications too.

Grinder

[petard]

For web apps, we've had a great deal of success with grinder [sf.net]. Its scripts are easy to write, and its recording mode is sometimes all you need to gin up a quick load test. For one project, we had Mercury as well... we found that grinder was much more helpful, at least for the development team. (Though I appreciated having both.)