Practical Software Testing Resources?

rhartness asks: "I've been a software engineer by profession for a few years and a programming enthusiast for much longer. As my experience has increased, so has the size of the projects that I have had to work on. My software testing method involves trying to do everything I can think of that the end user might try to do. Hopefully, this will break the application if there is a bug within my code. The current project that I am working on involves numerous tiers within a smart client environment. Trial and error testing is no longer sufficient — there is simply too much that could happen. Searching the Internet for software testing resources provides an abundant amount of information but it's often quite philosophical and verbose. What are some practical resources that Slashdot readers use for testing your software projects?"

automatic unit testing

[coyote-san]

A huge step is to put in unit testing / regression testing that's run nightly, and idealy with every build. You should at least cover the basics, e.g., at the persistence level can you create/read/update/delete a record? If you load a parent object, do dependent objects get loaded as well? At the presentation layer you can verify that a missing field will set the right error message.

You still need to do additional testing, but this will catch the underlying errors that can cause flakiness -- and worse, bad workaround -- at the higher levels in your code.

Automate

[Timesprout]

My software testing method involves trying to do everything I can think of that the end user might try to do

You need to automate your unit testing and let someone else perform functional and integration testing, prefereably qualified testers. Developers almost always test only positive flow which is totally inadequate when end users get their illogical mitts on the application.

Hire a tester.

[Bender0x7D1]

While full-time testers are often made fun of, ("Those who can't code, test."), or it is considered a secondary role that developers can perform on their own, there is no replacement for an experienced tester.

Testing is an important part of the development process. If you have an expensive and complex application, do yourself a favor, and hire a test team. Don't try to do it yourself, unless you have no choice. You may do an OK job of it, but it sounds like you don't have the experience to do it properly. This isn't a critique of your skills, it is a fact that you have a different skillset. You don't expect an embedded programmer to develop web apps or vice-versa. Could they do it? Sure. But it will be a long and difficult road as they make the mistakes an experienced person already known how to avoid.

If you have decided that comprehensive testing is important, (and some people decide it isn't), then do the right thing and get professional (test) help.

Re:My take

[jt2190]

...when requests come up (on an almost weekly bases) to change the code and rework specific modules entirely at the whim of management, things start to get complicated.

Willy-nilly changes are a huge risk to your success. Your management needs to focus on a goal, and let you reach it. It's like running twenty-five miles of a marathon and then having them move the finish line one you: Pretty soon, you drop dead from exhaustion. You can help them focus by making a list of all the stuff that they don't keep changing their minds about. Get that stuff built first. You can add window dressing later.

...as this project grows, I am reaching my mental limits of recalling and remembering what each method in that application does and why I even wrote some of them in the first place.

This is exactly why you want unit tests in place. The unit tests are your assumptions about how the code is supposed to behave, expressed in code. The ability to rerun hundreds or thousands of these tests very quickly against the code base, to make sure that things still work the same way as when you wrote them is a huge time saver.

By definition, unit tests don't test the assembly of the units, just the units themselves, so you'll still need to perform other types of testing to get complete coverage (funtional, integration, etc.) A good tester (i.e. someone who could write the program themselves but enjoys breaking code more than making it) can pay huge dividends here.

write a test plan, and/or hire a pro

[javaxman]

First, you should write a test plan. Actually, scratch that... first, you should write a design document. You have one of those, right? If not, you don't know what you're testing, and will eventually fail to cover some portion of your product, and that portion will likely break. Once you have a halfway decent design document, write a test plan that refers to your design document.

I'm going to second the notion I've seen in other posts that if your project really is big, there should be a programmer or analyst dedicated to testing it. You can hire someone who is less of a programmer and more of a tester if you want, but you'll get the best coverage if you can find someone with real programming chops who is able to design and implement tests. If you're doing your project right, that QA team will be every bit as involved in every portion of your project as you are.

Without knowing more about your particular project, it's very difficult to suggest appropriate tools. Different types of systems require different approaches to testing.

Verification?

[RAMMS+EIN]

As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took. I don't know if there are any verification tools for the programming language you are using (the ones I used were all for toy languages), but that was my first thought.

Re:My take

[Pootie Tang]

Given the additional information, I have a little more to say. It reinforces my impression that you are basically on the right track, there are just no silver bullets. However I think unit testing is probably more significant for you than I assumed at first. I think jt2190 expressed the value of unit tests very well.

I also agree with jt2190 about the significance of requirement changes. This is absolutely the area where you stand to make the most gains. If your boss does not have professional software development experience him/herself than he/she almost certainly does not appreciate how severe the consequences of changes are. Even if you tell them, they won't fully get it.

The fact that your boss is the brain child is a double edged sword, but mostly positive I think. Try to push more of the testing burden on your boss. After all, it's your boss who best understands how they want it to work. Explain that he/she can't just test the part that changed, but must as least give a cursory test to everything to check for side effects. That will help reinforce the importance of avoiding changes. Assuming your boss is as busy as most bosses are, it will also increase your chances of getting a dedicated tester hired (most likely from zero to slightly non-zero if you are the only dedicated developer).

When you get a new requirement or a requirement change, question it. Make sure you understand the underlying concern that is supposed to be solved by that requirement. This is an art, I'm sure you're getting better at it already. More importantly make sure THEY understand both the concern as well as the solution they are proposing. Specifying good requirements, just like good testing, is very hard and a lot of it comes from misidentifying the problem itself. If you have a good way of producing a mockup that allows them to explore the solution hands-on without needing to invest a lot of time to create it, it will go a long way. That's tricky as there are two closely related traps: the quick-and-dirty mockup evolves to production code without adequate planning or the quick-and-dirty mockup gets thrown out entirely despite the fact that some of it was valuable.

One thing I noticed you didn't say is that you are under pressure from your boss to reduce bugs. That leads to me to believe you realize how time consuming bug fixes can be and are self-motivated to find a solution. If so, that's a good sign.

One last thing that I should have mentioned initially as it applies to almost all situations, clean up that old code. It's inevitable you'll find crufty code as you go back to things. Sometimes it started that way for whatever reason (most often time pressure), sometimes it evolved that way. Cleaning it up (also known as refactoring) pays off 95% of the time. Unit tests are perfect when you want to change the code but not the behavior.

Re:Verification?

[ChefAndCoder]

Formal verification is one of those last vestiges (especially in academia) of the idea that it is possible to prove that software programs are correct. It's true for very small focused programs which do not have to interact with anything i.e. self contained. In practice, the effort required to formally prove that your software is correct is impractical. For "real" software systems, the number of unknowns introduced via API's, O/S and Hardware Interfaces, even just user interaction makes such "proofs" grow exponentially in complexity. If you are a software engineer, then you should realize that you will NEVER have the certainty of knowing something will work guaranteed every time. Formal verification is, frankly, a pipe dream.

Re:Verification?

[Raenex]

As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took.

In the real world very little software is formally verified. It's very expensive and time-consuming, and the vast majority of programmers don't know how to do it. Most projects just cannot afford it.

You're making too big a deal out of his "software engineer" label. There's no such thing as a software engineer, in the sense that for software there's nothing like the standards that engineers have to meet in other disciplines.