Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
The Almighty Buck IT

How Do You Prove Software Testing Saves Money? 312

Posted by samzenpus
from the taking-it-for-a-test-drive dept.
cdman52 writes "I work at a small software development company. We have one app that is used by a few hundred clients and was initially developed by a few undergrads about 10 years ago. The app is collection of about 25 developers preferences and ideas. Testing wasn't an initial concern since it was created as an internal application, I guess. Anyway, the app is now large and used frequently. Part of my duties are to fix bugs users find, I'm on a team with a few other people and at least once every 2-3 months I see some bug I fixed come back, and I can only assume it's because we don't have a formal test suite. The owner doesn't want to invest time or money in getting one set up, but I'm sure that in the long run it would save time and money. Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?"
This discussion has been archived. No new comments can be posted.

How Do You Prove Software Testing Saves Money?

Comments Filter:
  • by ls671 (1122017) * on Monday January 03, 2011 @03:44AM (#34741948) Homepage

    I would say the most important thing to save money is a good design and programming guidelines for the application.

    Test suites are obviously a good idea but it may end up as a tedious task if the application as poor or no real starting design and if everybody just did things their ways.

    I have seen many applications that had grown larger than expected that typically cost 10 times more than it should to maintain, bug fix and implement change requests.

    It is sometimes possible to fit existing code in a new clean design and that saves a lot of money if the application is going to be used for a long time.

    Let's hope your application has proper design ! ;-)

  • First things first (Score:5, Insightful)

    by Anrego (830717) * on Monday January 03, 2011 @03:46AM (#34741954)

    Can anyone offer suggestions for how to convince the owner that setting up a test suite is in his own best interest?

    Are you sure that it will be?

    but I'm sure that in the long run it would save time and money

    How much time and how much money?

    What you need here is hard numbers to back up your opinion. Specifically, and I hate to say the word, but you need metrics.

    - How much do these bugs cost (these are a mixruee of hard numbers (dev time, etc) and "fuzzy" numbers (customer relations and so forth)?
    - How much is setting up this test suite going to cost? How much is it going to cost to use it?
    - How effective do you think it will be (look at existing bugs.. would they have been caught)?

    That last one is especially important. Testing is good, but it doesn't solve all problems and there are other options. Really take a good look at whether previous bugs would have been found in your proposed system.

    Once you have this information, the math then becomes pretty simple.

    If your program is modular-ish .. it might be possible to do a trial. This is probably a good idea anyway. There are different ways to do testing, some of which are incompatible with certain development mindsets. Best to find that out early on a small piece vice the entire code base.

  • by SuperKendall (25149) on Monday January 03, 2011 @03:57AM (#34741982)

    You need to think very hard about exactly how any why this will save the business money. What do YOU think will be gained, for your software, from a technical standpoint?

    Many people just think testing will make things better. But I have seen testing make things better, and I have also seen testing make software far harder to maintain and later to deliver.

    What kinds of problems are you seeing that you think testing will solve? Do you see a lot of maintenance causing other errors? Do you want to do this in preparation for overhauling some part of the system?

    You see bugs come back every 2-3 months, but is preventing that really worse than the overhead of maintaining and adhering to a test suite that all of the developers would have to do? That alone may not be enough of a reason to impose the overhead that testing can cause.

    Since you are fixing bugs you have access to the codebase. Start a test framework yourself, of bugs you cover, and run it against the code day by day. See how often it catches issues, vs. how often you have to alter the tests because the code has changed in ways that are valid. Then you will have more real world metrics about problems solved vs. costs...

  • start small (Score:5, Insightful)

    by sugarmotor (621907) on Monday January 03, 2011 @04:00AM (#34741986) Homepage

    Just start small, with "failing tests" for each new bug. Bug fixed, test passes. Keep expanding the test coverage.

  • by initialE (758110) on Monday January 03, 2011 @04:01AM (#34741992)

    Perhaps the app is just more important to you than it is to the company? First take your ego out of the equation, i guess, then make your case before the boss. Just sayin'. Also if you have a time sheet of how much time you've spent on this app so far it would be helpful in drumming up some numbers, plus a few scenarios where the failure of the app causes lost time for the company as a whole.

  • by Rosco P. Coltrane (209368) on Monday January 03, 2011 @04:19AM (#34742066)

    WHY do you have to prove software testing saves money? Even a cheap and nasty electrical appliance from China is tested despite there being a lot of motivation to cut corners and not do so. Why should software be any different?

    Rest assured that Chinese companies test their products precisely *because* it saves money. If it didn't, they wouldn't; they don't have a commitment to good quality for its own sake.

    An untested product leads to high rates of return (lost money), customer dissatisfaction and brand name damage (HUGE loss of money in the medium to long term). Nobody puts up with bad products anymore. Software is one of the last kind of products where it's still somewhat accepted but people are quickly becoming intolerant to bugs nowadays.

    Still, what is true of physical products (that extensive testing on top of proper design and good manufacturing practices) may not be true of software. I.e. the question is: is laborious and careful design and implementation with minimal testing more or less expensive than quicker, laxer design and coding and a strong test/correction feedback process? I'm not sure the answer is clear-cut. As a (former) programmer, I can see argument for both approaches.

  • Re:start small (Score:5, Insightful)

    by VirexEye (572399) on Monday January 03, 2011 @04:32AM (#34742098) Homepage

    +1 to moding parent up.

    You won't get far convincing a product owner that you should spend months writing tests for the entire system.

    Convincing someone that you should write unit tests for all new functionality to help guarantee the bug fix/new feature will continue to always work into the future is a much easier sell.

  • by SanityInAnarchy (655584) <ninja@slaphack.com> on Monday January 03, 2011 @04:35AM (#34742112) Journal

    As others are saying, if you can get hard numbers, get them. Nothing like actual evidence to back up your claim.

    However, there's a more important reason for doing this which is harder to find solid evidence for -- the concept of "paying down your technical debt." Often, we hack software together relatively quickly, without fully testing, etc -- every time we do that, we're effectively borrowing against the future stability and maintainability of the product, or realistically, against our own debugging and refactoring time in the future.

    Like all debt, it has interest -- the longer you let it fester, the worse of a mess it's going to be. So like all debt, it makes sense to pay it down when you can -- a little refactoring now can save a lot of headaches later.

    So, it may be difficult to get hard numbers -- though again, those are better. But if you can't do that, sell it as an investment. Invest a little time right now at least putting the automated tests in place, and maybe start with just some regression testing. Twenty minutes writing a regression test will almost certainly save you an hour the first time that bug appears -- and those are pessimistic numbers (though also made up on the spot -- find your own actual facts!)

  • by mcvos (645701) on Monday January 03, 2011 @05:11AM (#34742216)

    Well, in my experience it goes like this: You're looking to refactor the code because the code already is quirky. Some 99% of the time, that's just because it's sloppy coding, poor structure and logic or it is simply undefined or irrelevant, the corner cases are never reached, the function is always called with data of the correct format and so on. Then in 1% of the cases, the application depends on this quirky behavior and all in all works correctly - or at least works - until you clean the code. If you have to assume that everything that is there should be there, you end up with code almost as bad as what you started with. This is the problem trying to reverse engineer the intended behavior from existing code.

    1%? I think even if the code is buggy, it's quite likely that the code now depends on those bugs in various ways.

    In any case, the right way to refactor is to make sure you've got proper tests for the code you're going to refactor. That way, after refactoring, you can check whether the functionality is still unchanged.

    So yeah, the submitter has a problem. What he really needs from a code-quality point of few is pretty big and expensive. If there's no money for it, but there is money for continuing as before, then I guess that's the only option here.

  • by Kjella (173770) on Monday January 03, 2011 @05:20AM (#34742236) Homepage

    Still, what is true of physical products (that extensive testing on top of proper design and good manufacturing practices) may not be true of software. I.e. the question is: is laborious and careful design and implementation with minimal testing more or less expensive than quicker, laxer design and coding and a strong test/correction feedback process? I'm not sure the answer is clear-cut. As a (former) programmer, I can see argument for both approaches.

    Didn't you just answer your own question: extensive testing on top of proper design and good manufacturing practices

    For the first I'll just quote Knuth: "I have only proved it correct, not tried it." And if you send piles of shit to testing/QA and expect them to be the impenetrable barrier between you and the customer, you're equally deluded.

    I've found that for anything that should last a while and be stable, there should be a test case. It's so easy to subtly break something. However, I've found writing a proper test suite that deals with databases, network communication+++ and not just the application itself is pretty hard.

  • by Splab (574204) on Monday January 03, 2011 @05:42AM (#34742304)

    Also, putting testing infrastructure into an existing system is a huge undertaking. And it requires a very strict handling for all future commits - if you just slip a little on the test suite you are basically invalidating the whole thing.

    Many a times I've tried to implement test suites, both as request from employers and also for own sanity sake - and every time it has failed because the workload was simply too overwhelming to support both testing and development.

    In the end, the basic economics of it has dictated that it is cheaper to hot fix than to make it right the first time around.

  • by dtmos (447842) on Monday January 03, 2011 @06:44AM (#34742478)

    If you truly are working for a small software development company, cost may not be the issue. Frequently (I may even say universally) the issue facing the owners of small companies is not cost, or even profitability, but cash flow -- making sure that there is enough money coming in to make the next big expense coming up (frequently payroll). In small companies there typically aren't the cash reserves available to spend on unanticipated expenses or program delays. The boss may even agree with you that the overall cost would be reduced if software testing were introduced, but may not "want to invest the time and money" because the company does not have the cash available to make the investment.

    For example, the software you're working on needs to ship by the end of January to receive payment from the customer by the end of February, so that payroll can be made in March. If bug-fixing stops in January for development of the bug-fixing program, the customer doesn't get his software until the end of February, so he doesn't pay until the end of March and so payroll is missed that month. Having fewer bugs in the long term has to be balanced by the need to stay in business until the "long term" is reached.

    It's like anything else in finance: You have better options and get a better return on your investment if you invest $100,000 than if you invest $1,000, but if you don't have $100,000, it doesn't much matter -- you do what you can with the resources you have available. Similarly, you get a better return if you're able to invest your money for a year than if you must have it back in a week. (The guy who said "time is money" actually knew something.) This trade between cash flow and long-term efficiency (leading, one hopes, to profitability), i.e., knowing when and how much to invest, is the real art of business management.

    The solution to your problem? IMHO, incremental investment. I agree with those above who suggest implementing tests for new bugs found. This should enable you to begin to work more efficiently over time, without substantially delaying current work, while collecting metrics (I know, I know...) that can help quantify the cost of the bug re-work. Once a substantial body of tested bugs has been collected, one can institute a formal testing program (preferably by including it in the next project, so that it is a planned expense) as a cost reduction, since by then it should speed up development work over the ad hoc method then in place.

  • by gbjbaanb (229885) on Monday January 03, 2011 @07:17AM (#34742590)

    Software is one of the last kind of products where it's still somewhat accepted but people are quickly becoming intolerant to bugs nowadays.

    in consumer devices perhaps, but that's a different area to most software. For most software sold, especially to business users, bugs are accepted as one of those things that happen. Now, really buggy software will eventually get replaced but in most cases the important thing is to have a good (or excellent) customer service that responds to, and fixes the bugs. If you have that, your customers will not care so much about bugs that they find. you just need to be responsive, and considerate in your communication with them. Once you have that in place, you'll often find that its an asset to you, the more bugs the customer finds, the more they feel loved. And customers like feeling that you care for their business - especially when its some executive/supervisor reporting them, and some grunt who has to actually use your PoS software. Its how the big names can make such poor enterprise software and still have it selling in billions of dollars a year in maintenance/support payments.

    I think I might have just explained how buggy software is good for your business. I feel dirty.

  • by Glonoinha (587375) on Monday January 03, 2011 @07:38AM (#34742648) Journal

    In the end, the basic economics of it has dictated that it is cheaper to hot fix than to make it right the first time around.

    What the OP failed to provide is the nature of the bugs his users are experiencing, the impact to the users and to the business, and the costs associated with the manifestation of those bugs.

    If we're talking about user interface glitches where a button is not aligned with the others, the color of the button when pressed is wrong - then it is probably way cheaper to hot fix than to insure 100% quality the first pass.
    If we're talking about workflow and data errors in a financial application with regulatory implications, or bugs that bring the system down for an hour and the company loses five figures in revenue for each hour of downtime, that's a different story.

    Testing doesn't necessarily save money. Saves pride and credibility more often than not, but as others are saying it isn't necessarily the most fiscally prudent choice. Neither is a wholesale re-write of 10 year old code (even though that's our first inclination when we inherit code.)

  • by Lumpy (12016) on Monday January 03, 2011 @08:00AM (#34742736) Homepage

    Exactly, the cost of not testing may be a IT helpdesk position that can be eliminated by doing better software testing. It can also be loss of productivity, loss of work quality, etc....

    Honestly, just because it's software does not make this hard, replace the word software with "business process" or "machine" and you have the exact same problems. Would a company buy a $250,000 machine without research or testing to see if it was the best fit for their solution? No.. yet many buy the crap Ticket system that is BMC Remedy for close to the same price just because everyone else does.

    testing is not a IT process it's a business process. Lack of software testing is simply a indicator of a failure of the management in the company.

  • Lose The Job (Score:5, Insightful)

    by assertation (1255714) on Monday January 03, 2011 @08:19AM (#34742808)

    If you are catching shit when a bug happens and they refuse to set up formal testing get out of the job now.

    If you aren't catching shit of any kind, then look at the bugs as helping you keep employed. If it annoys you, go find another job.

    I've worked for similar cheapskates. They aren't likely to change.

"Someone's been mean to you! Tell me who it is, so I can punch him tastefully." -- Ralph Bakshi's Mighty Mouse

Working...