Suggestions for Performing Regression Testing? 37
gmletzkojr asks: "The company that I am currently working at develops a fairly complex industrial controller, complete with embedded software, a GUI on the controller, and a Windows app to connect over Ethernet. On previous versions of a similar project, we have performed testing manually - ie, monkey presses button, and sees that light turns on, widget turns, GUI updates, etc. However, this is extremely time consuming (previous complete regression testing took ~3 weeks) and is error prone in itself. How do you perform complete system regression testing? Do you use shrink-wrapped packages, or build your own? How do you test features that are easy for humans to observe, but not as easy for software to detect (ie, the light came on, the GUI updated when I pressed the external input, etc)?"
Well, I did regression for a big company... (Score:3, Insightful)
Anyway, doing regression testing is a 2 edged sword.
1. You have to program more stuff into your program to allow for automation. Now, because you're adding another feature to the entire program, that means more testing to ensure that the added functionality actually works. Lastly, you have to hire/write regression scripts. How much time does it take to write scripts? How long will they be useful for? If it takes ~the same time/money to write scripts as to hire/be the monkey...How much is that worth?
2. It can be a big convenience if you can justify #1.
Grump
Re:Well, I did regression for a big company... (Score:2)
class Foo
{
friend class FooTester;
};
That's it. Now you can do whatever you want in your regression testing without molesting your existing code (having to "program more stuff into your program," as you say)
Break down the problem and test components. (Score:5, Informative)
How do we test this you ask?
Break down the problem into:
(a) components
(b) component integrations (i.e. network/middleware checks)
Automate tests for each and you're at least 80/90% of the way there. I.e. Unit test pieces such as a UI or Middleware, and also make unit tests that verify the connections between components.
If you're unlucky enough to be in a totally mixed-up environment without any seperation between GUI, logic, back-end, etc... often lumped together as MVC, but I prefer just to say "component based" you may have to do some re-architecture to get to a better scriptable setup.
Our automated tests (web environment, mind you) test for:
1. Basic web server & Tomcat / Servlet function.
2. Web/Tomcat Server -> Middleware connection (no actual request)
3. Tomcat->Middleware->Legacy System connection.
4. Tomcat->Middleware->Legacy System operations - i.e. actually do something.
Given the complex and ever changing data we operate with, we can't test everything that only a human eyeball can spot. However, these tests are re-used for live system monitoring, regression testing, and load testing. They also can find many of the "brain dead" issues - such as a total system meltdown or specific function being broken by a code change without requiring a tester to test everything out.
I strongly recommend going for automated testing as much as possible - even if you're only getting at the low hanging fruit. Breaking down the problem into managable chunks w/unit tests can also save serious hassle trying to figure out later why a tester got an error somewhere.
Re:Break down the problem and test components. (Score:2, Insightful)
Answer: I work at a company with a 30+ year mainframe application that runs exclusively via line mode commands followed by several paragraphs that have nothing to do with the original question, but instead tell how to use (yawn) unit testing.
Relevancy: None.
Re:Break down the problem and test components. (Score:1)
Re:Break down the problem and test components. (Score:2)
He could have connected the dots a little better, but he described a perfectly good approach. The poster asked How do you perform complete system regression testing? and a good way to do that is to break the system down.
Note that the post you're griping about talked not just about unit testing, but integration testing. Both are important to getting an app like this well
Re:Break down the problem and test components. (Score:1)
noahbagels' post laid out the process which is followed where he works, which addressed both the time consuming and error prone aspects as well as answering the first question. In noting specifically how they've gone about doing it, he answered the second question.
You, of course, have tried (unsuccessfully I might add) to show y
Is there an Open Src ver of MS Test? (Score:2)
A few years ago, MS had a testing aid called
- creatively enough (for MS, at least...)
Microsoft Test.
As you didn't even get it with MSDN Universal
I never got the chance to evaluate or use it,
but perhaps there is a counterpart in OSS...?
I'd love to find one... TIA
Re:Is there an Open Src ver of MS Test? (Score:3, Informative)
Re:Is there an Open Src ver of MS Test? (Score:1)
Judging by your UID, you are not new here... but let me tell you how times have changed. Microsoft now advertises on Slashdot!
Yes my friend. The distribution of Linux users on this site is swiftly declining. As Slashdot becomes more mainstream, you can expect to see this trend continue.
Mercury? (Score:4, Informative)
They have Quality Center:
Which will help you write manual test scripts and create new test cases and do defect tracking.
Quick Test Pro:
Semi decent intergration into quality center so you write your test cases, then bust it out to code. You CAN just simply record a test and run it, but the lifetime of your script may vary. You need to build in tons of logic just to ensure the script can handle what it thrown at it and handle various error conditions.
Caveats:
If you like open source, this is not for you.
If you don't like vbscript, you may find a difficult path to using other languages. There are options, but if you have a large automation team, a common language is useful. Vbscipt is sometimes the common denominator.
If you ARE familiar with Load Runner or their performance tools, you'll love it.
Even if you don't mind vbscript you may find their tool unweildy in the way it picks up objects.
All in all, probably the BEST commercial offering - simply because a monkey can use it. There are good ones in opensource if you can code, i.e. "Push To Test" if you like java.
ps: sorry there are no links, I would just google them anyway
Re:Mercury? (Score:1)
Re:Mercury? (Score:3, Informative)
I have built testing frameworks in both WinRunner and QuickTest Pro, and I have to say the QTP framework was easier to write. Using the XLS-as-database structure built in to everything COM, we were able to write tests in Excel files, and do it in about 1,500 lines of code.
Anything is simple if you break down each step into a window, object, action, input, output format. One thing we found that makes our lives somewhat easier is to forego the entirety of QTP except as an execution engine. We don't use actio
Re:Mercury? (Score:2)
Separate IO from the guts of the system (Score:4, Insightful)
A stopwatch? (Score:1)
AutoIt? (Score:5, Interesting)
Looks like the free AutoIt [hiddensoft.com] would help.
Also see AutoIt Script home page [autoitscript.com].
Re:Much more importantly... (Score:3, Informative)
Re:Much more importantly... (Score:2)
This can happen when developer A fixes a bug, and developer B either overwrites the change or assumes developer A made a mistake turns it back to the way it was.
Regressive bugs are often the product of a failure in team communication or a weak development process; idealy you'd never have any regressive bugs, but this ain't a perfect world, and I ain't
Good luck... (Score:5, Informative)
Good luck. I have found that regression tests often cost just as much as just paying someone to do the test. The only time they are an advantage is when you run them against the daily builds. The problem is when you do that every day one test will fail because of a new feature, and the test now needs to be changed, not a bug written up. (Of course you will catch bugs too) So you have to have a full time programmer who only job is to keep the regression tests up to date. This can be more expensive than just hiring testers. (Kids are cheap and most of them can operate a computer well enough to do your testing if you keep track of them)
Don't read too much into that downside. The more expensive programmer will in the end catch more bugs with the regression test suite, so your code has less bugs when you ship. However it is expensive.
Automatic regression tests are NOT a substitute for manual regression testing! You still need to plan on some time (though not the 3 weeks you did before) for regression testing, just to make sure the automatic tests didn't miss anything. Much of this will be done as part of your normal tests, but make sure you are watching for regressions when you test.
Re:Good luck... (Score:2, Interesting)
I think this can be true if you write your regression tests afterward. But if you're doing test-driven development, so that your app is designed from the beginning for testability, then I think the costs are pretty reasonable. In my experience, the time testing is more than made up for by the time and stress saved on debugging.
On a recent 9-month
Re:Good luck... (Score:1)
If you are thinking of automated testing... (Score:3, Insightful)
Also, people tend to think that automated testing takes less time...it *CAN* though expect that on many projects it will take much more time as automated tests are detailed and implementation specific; you can't create tests at the spec level unless your specs are detailed design documents too and even then only in a limited way.
The time savings kicks in when you want to frequently repeat the tests across the whole project when even minor changes are made to the code in one place. It also allows you to be somewhat certian that only the things you expect to change do indeed change.
If you do not have the budget or time to do complete manual tests, forget about autmating it unless you are dealing with a very static project that requires excessively detailed testing.
I expect people to disagree on much of what I wrote above...when they do, keep in mind that situations can differ. These are just general rules of thumb and worked for the vast majority of projects I've been on.
Re:If you are thinking of automated testing... (Score:1)
I am a software test engineer, I lead a small software test team, and am a huge advocate for automation when it is the best solution to the problem.
James Bach, a leading test automation consultant, wrote an excellent article [satisfice.com][PDF] debunking some of the reckless assumtions folks make about test automation.
Generally speaking (and this is VERY context-dependent) I would like to have both skilled manual regression testing and regression suites written by skilled automation engineers, but if forc
Testing physical devices (Score:5, Insightful)
For the GUI, I recommend instrumenting your app so you can programatically tell what's going on. An API is one way, but a quick and dirty way is just to keep an internal event log and then probe that. Then for free, you get a detailed log you can dump if there's an error in production.
For the physical hardware, consider building a simulator. You could do it partly in hardware, adding simulation logic to your hardware controller, but running disconnected from the machinery. Or you could build another board that connects to your production board's inputs and outputs and simulates the machinery at an electrical level. Another option is to simulate your production board entirely, leaving the embedded code out of the testing loop.
The right choices depend a lot on where you can get the best bang for your test automation buck. Unfortunately, starting with a lot of untested legacy code means you have a long slog ahead of you. Start with the modules that generate the most bugs, or the bugs hardest to find during manual testing and automate your tests of those. That will teach you a lot about good ways to test in your environment.
Re:Testing physical devices (Score:2)
There are also tools that query the actual components rendered on a window - sort of like the Spy++ tool included in Visual C++. This is the approach taken in the book "Effective GUI Test Automation" (ISBN 0-7821-4351-2)
With cheap webcams, open-source software for detecting/comparing changes in images (sim
Re:Testing physical devices (Score:1)
This is true. If you're going for theoretical purity, that might be necessary. In a commercial effort, though, you're more interested in bang for the buck.
If you instrument your controller code (which is often easiest) then you catch all bugs except for GUI toolkit ones and in connecting the GUI library to your app, which is, I understand, not a big source of bugs, as it's simple stuff that programmers tend to hand-test pretty well.
If you go further an
Re:Testing physical devices (Score:2)
I suggested the webcam for the physical blinking lights stuff the poster mentioned - this should be possible even today with open-source surveillance software I mentioned in passing (set zone, and the software continually analyse for image differences). For the GUI, screen capture would be more efficient.
True, creating a testing tool that did screen capture would be a hassle. However, IBM already Also, most informed people also
Mercury Interactive Toolsets (Score:2)
Mercury Interactive makes a great set of tools for regression and load testing. If the app is Windows based, WinRunner or QuickTest will allow you to script end-user actions for repetitive testing scenarios. LoadRunner is a great way to test a back end system, not just for load but also regression. And, if you're not useing a bug-tracking mechanism already, TestDirector is a nice web based tool for bug tracking that integrates well with the other tools.
I u
For Java Swing, try QFTest (Score:2, Informative)
It is similar to many of the other products that do automated GUI testing, but will run on Linux and Windows (others are very Windows centric). It is also is scriptable using Jython.
Doesn't relate directly to your needs but I thought others working with Java might find this information useful.
JUnit (Score:1)
Re:JUnit (Score:1)
A good unit test suite will catch many errors which might break existing functionality. Regression testing on the other hand tests changes after integration, tends to be more GUI-driven and user-oriented, is often best done manually, and is likely to find a somewhat different set of bugs than unit testing will.
GUI, hunh? (Score:2, Insightful)
Rational TestStudio (Score:1)
Other than having to give up your left testicle to afford it (I volunteered my boss for it), it proved itself quite capable of handling unit, integration, system, regression, and full-bore load testing. If you're also planning to do any load testing the "virtual users" will cost you the other testicle (I'd suggest you use multiple levels of management's PHBs - they get skittish after a couple of roosters become hens).
Different kind of regressions (Score:1)
I problem we have today it's a conflict between R&D testsuites (corner cases, basic functio