How Do You Prove Software Testing Saves Money?

cdman52 writes "I work at a small software development company. We have one app that is used by a few hundred clients and was initially developed by a few undergrads about 10 years ago. The app is collection of about 25 developers preferences and ideas. Testing wasn't an initial concern since it was created as an internal application, I guess. Anyway, the app is now large and used frequently. Part of my duties are to fix bugs users find, I'm on a team with a few other people and at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite. The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money. Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?"

Cost-Benefit Time!

[wild_berry]

We've had to justify creating auto test suites where I work.

Over the last decade our product has grown from one code-base into three strands, each with separate customer foci, and we've had a healthy amount of staff turnover so that there are still brilliant, creative and skilled people working on it but some of the original knowledge has left us.

We found* numbers to justify that automated testing of existing features, applied later will protect against regressive changes. Even where there are complicated features which were not modular in design, or which lack good interfaces, the tests have saved us massive amounts of time testing by hand. The real win is hidden under something we didn't realise until later: creating the tests have forced us to really document what the features are and how they work**, sometimes from a unit-test level, sometimes at the interface level and sometimes in a top-to-bottom vertical slice. Once you have a record of what your software does, in a computer which is skilled at remembering exactly and repeating exactly what some former staff member told it a couple of years ago, you have a decent reason to be confident that your bug fixes won't cause more harm than good.

*: ballpark figures / educated guesses / made up.
**: We favour working code over comprehensive documentation, until our agile team is reassigned to other projects or leaves the company.

Eventually you need a solution

[nanospook]

The metrics is a good idea. This will allow the owner to see what it costs him to not have good QA. However, eventually, the conversation will float to how expensive will it be to have QA? Like any security situation, you will want to find the happy medium where the trade off in time/money balances the risk. Given you only have a few hundred users you might wish to experiment by adding QA incrementally. Also, you might find why those bugs are coming back? Poor documentation? Or are developers not maintaining up to date code bases? Process is just as important as QA when it comes to fixing bugs.