Web Performance and QA Tools? 25
perf_monkey asks: "I'm part of a large Web Infrastructure Quality Assurance (QA) team at a large financial institution. Currently, we use Mercury Interactive's LoadRunner like a bunch of trained monkeys. We also use QA Load and SilkPerformer, but for smaller (non-J2EE) projects. As one of the technical folks, I've been trying to expand our horizons and our budget. I can no longer believe that large companies are willing to pay a QUARTER OF A MILLION dollars for the privilege of an additional 2000 Mercury VUsers. I'm looking at both commercial and open source alternatives. I've been tinkering with The Grinder and have had pleasing results. While not a full-blown QA tool, it is an excellent 'programmer's' load test tool. I was hoping that there are other tools like this and was hoping for the community's opinion. What web performance tools do you use and what do you think of them?"
Under windows? (Score:3, Interesting)
Re:Under windows? (Score:2)
Question (Score:1)
How does lode runner test systems now?
I work (Score:2)
Now, I'm not saying the spending isn't wasteful. I think it is, but it's not correct to say that large companies are paying for expensive SQA software licenses anymore.
Perl and Java (Score:1)
Perl and Java are your friends...
Web Performance Trainer vs. OpenSTA vs. Mercury (Score:3, Informative)
I've used Mercury products for unit testing, full out scaling tests, monitoring, and defect tracking for over three years. The problem was the total cost of a Mercury load test bed. It gets expensive rather quickly unless you have planned in advance or do a lot of Mercury work. If you don't it makes more sense to hire a consultant that has an open license arrangement. That way you only pay for when you use it. YMMV.
So, I looked into some other alternatives and ended up going to unit test applications from the Apache group and home grown benchmarks to approximate load scenarios.
Eventually, I ended up looking into OpenSTA OpenSTA [opensta.org] and was fortunate enough to work with some real experts that did a lot of load testing work with OpenSTA. There are some limitations to the OpenSTA *cough* Windows only *cough* but overall it is an excellent tool for driving load for a complex web application. Much like Mercury or any load testing tool the test is only as good as the planning and analysis you perform to isolate performance issues.
My most recent job called for load testing but in that time Mercury changed their license program again and we really wanted to use something that would run on Linux. So, OpenSTA was not really in the running. Also, we needed to simulate multiple IP addresses not just hundreds of virtual users from a single source address. (fine for simulations of a access from a corporate firewall). So, I had to find something that would work and that wouldn't blow our budget.
I checked back with a company called WebPerformance Inc [webperformanceinc.com]. Now, I looked into a few years ago to see if they supported IP spoofing for virtual users and SSL. They had done SSL but the IP spoofing wasn't done.
As it turns out, they put this feature into their 2.6 release. We use it to run our large tests for burn in and acceptance for revisions to our network hardware that provides web interface. Like Mercury, you can really ramp up serious traffic. Couple this with basic network load generation and you can create a sound simulation of network throughput and application access from multiple network addresses with mulitple users and multiple business cases. Oh, and the fact that you can get a price list that is straight forward is very nice.
So, I'd say each is right for a certain type of test and a certain type of shop. For our shop, Mercury is cost prohibitive, OpenSTA lacks a key feature (IP Spoofing), and WebPerformance, Inc -- while commercial -- satifies our feature requirements near perfectly.
Open STA? (Score:1)
Re:Open STA? (Score:2)
Windows only (Score:2)
Re:Windows only (Score:2)
It's not impossible just slightly cumbersome.
Web Performance Inc. (Score:1)
http://www.webperformanceinc.com/
They've been quick to respond to our concerns, and their price was significantly cheaper than any of their competitors.
Load Generators (Score:2)
Save you money (Score:2, Insightful)
Solution such as LoadRunner were too expensive, but even renting LoadRunner users for a few days was too much. We figured that for the money we saved by not checking the load we could buy and host two additional servers f
use curl (Score:1)
OK this is not comprehensive and slick but it worked sufficiently to prove we could handle 100 c
Re:use curl (Score:1)
Reduce the scale rather than increase the licenses (Score:2)
We had a limited budget, and buying even tens of thousands of dollars worth of licenses was - in our view - a bad use of that budget. Instead, we concentrated on building a test infrastructure with known performance characteristics, and extrapolated (sorry, I meant "guessed") that this performance profile would scale by a factor of X on the live infrastructure. By using a relatively small and flexible
Uh. (Score:2)
Open Source Testing Tools (Score:1)
Home-brew automation guru (Score:2)
Custom HttpUnit code (Score:1)
The main reason testing existing applications is so hard is that they aren't designed to be easily tested, so you have to jump through a lot of hoops to do proper testing. However, if the developers don't get to check anything off until their components are covered by unit tests and their features are covered by acceptance tests, that gives the developers the incentive to mak
Re:Custom HttpUnit code (Score:2)
A/k/a "requirements driven development", among other names. It's generally considered to be a sound development strategy for any kind of software development. First you figure out -- if only roughly -- what the software needs to do, then write tests that will verify that a given implementation will work, and only then do you start working on the actual system itself.
Once development is under way, the goal should always be to make it correct first -- and you know it's correct because it passes the tests (p
Easy! (Score:2)
Repeat and tweak as necessary.
THANKS for your help (Score:1)
I just wanted to say thanks for everyone's suggestions. I consider myself a pretty experienced perf expert and I learned about some good OS tools. I'd like to give a bit more of a background of my setup.
My PTE (Performance Testing Environment) is made up of an AWEFUL lot of REALLY expensive Sun boxes. That is where I use Mercury, and will continue to use Mercury. I have another setup for the DIT, SIT, and DEV environments. The load generators that I use in the aforementioned environments are 5 beefy x86 R