Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
Programming IT

Practical Software Testing Resources? 42

Posted by Cliff
from the verification-as-important-as-creation dept.
rhartness asks: "I've been a software engineer by profession for a few years and a programming enthusiast for much longer. As my experience has increased, so has the size of the projects that I have had to work on. My software testing method involves trying to do everything I can think of that the end user might try to do. Hopefully, this will break the application if there is a bug within my code. The current project that I am working on involves numerous tiers within a smart client environment. Trial and error testing is no longer sufficient — there is simply too much that could happen. Searching the Internet for software testing resources provides an abundant amount of information but it's often quite philosophical and verbose. What are some practical resources that Slashdot readers use for testing your software projects?"
This discussion has been archived. No new comments can be posted.

Practical Software Testing Resources?

Comments Filter:
  • release it! (Score:4, Funny)

    by Hell O'World (88678) on Thursday November 30, 2006 @12:51PM (#17051824)
    Release it with a big BETA label. Let your customers find the bugs.
  • TDD, JUnit (or equivalent), FIT. Well-tested, independent tiers. GUI test as a last resort, or with humans.

    --Robert
  • by faster (21765) on Thursday November 30, 2006 @01:11PM (#17052166)
    Read "The Art of Software Testing" by Glenford Myers. Even if you only get a few chapters in, you'll get a lot of ideas that will improve your testing. "Testing Computer Software" by Cem Kaner is another good one.
    • by rhartness (993048)
      Does anyone else have any thoughts on "The Art of Software Testing"? It appears that there was an update to the book in 2004-- the first since it's initial publishing in 1976. Is it worth getting the second ed. over the first?
  • by coyote-san (38515) on Thursday November 30, 2006 @01:12PM (#17052172)
    A huge step is to put in unit testing / regression testing that's run nightly, and idealy with every build. You should at least cover the basics, e.g., at the persistence level can you create/read/update/delete a record? If you load a parent object, do dependent objects get loaded as well? At the presentation layer you can verify that a missing field will set the right error message.

    You still need to do additional testing, but this will catch the underlying errors that can cause flakiness -- and worse, bad workaround -- at the higher levels in your code.
  • Automate (Score:3, Insightful)

    by Timesprout (579035) on Thursday November 30, 2006 @01:12PM (#17052184)
    My software testing method involves trying to do everything I can think of that the end user might try to do
    You need to automate your unit testing and let someone else perform functional and integration testing, prefereably qualified testers. Developers almost always test only positive flow which is totally inadequate when end users get their illogical mitts on the application.
  • by Z0mb1eman (629653) on Thursday November 30, 2006 @01:18PM (#17052322) Homepage
    Step 1: write a systematic test plan. There are all kinds of acronyms and books out there on the subject - I doubt you need it. At the very least you can start with CRUD (Create, Read, Update, Delete) and go from there. Cover the success path, cover the failure paths.

    Step 2: automate. This isn't optional if you're planning to maintain a project beyond the 1.0 release. For specifics, it depends on your project. Is it mostly a Java app (http://www.junit.org/)? Is a lot of the logic in the database (http://www.dbunit.org/)? Is it a web app, regardless of language (http://webtest.canoo.com, http://wtr.rubyforge.org/ [rubyforge.org])?

    Step 3: run your automated tests and laugh at how much easier it makes your life than manual testing.
    • by ygthb (84559)
      Step 1: write a systematic test plan. Cover the success path, cover the failure paths. Gather all your data sources

      Step 2: automate. follow the above advice if you want to create our own harness. But for an easier way check out the commercial tools (trying to be non-blatant, i am a vendor, check my profile)

      Step 3: run your automated tests and laugh at how much easier it makes your life than manual testing.

      An update to the previous poster.
      • by Z0mb1eman (629653)
        >follow the above advice if you want to create our own harness. But for an easier way check out the commercial tools

        Commercial tools are definitely worth evaluating, yes, but as a programmer, I have yet to find one that actually makes my life easier (let alone be worth the usually exorbitant prices). They're great for places like banks, which might have a large budget and relatively low technical expertise, but (generalizing here) they don't sound like what the poster is looking for.

        • by ygthb (84559)
          Z0mb1eman,

          It is up to the individual development group, but getting the fox out of the henhouse is as important as developing a test plan for accurate results. Programmers are taught to think in a certain way, that is not a bad thing. I can always tell when a programmer developed the test plan, or the subsequent tests. This is not to say that it is all bad. Programmers make great unit testers. But when it comes to wholistic testing, and what your users are going to subject your application to, a pro
    • Re: (Score:1, Informative)

      by Anonymous Coward
      Step 1: write a systematic test plan. There are all kinds of acronyms and books out there on the subject - I doubt you need it. At the very least you can start with CRUD (Create, Read, Update, Delete) and go from there. Cover the success path, cover the failure paths.

      Step 1.5: show the test plan to someone else who's reasonably familiar with the project being tested. Let them ask questions about the test plan. Add any new test cases they think of (or that their questions cause you to think of) that you f
  • Hire a tester. (Score:3, Insightful)

    by Bender0x7D1 (536254) on Thursday November 30, 2006 @01:22PM (#17052386)

    While full-time testers are often made fun of, ("Those who can't code, test."), or it is considered a secondary role that developers can perform on their own, there is no replacement for an experienced tester.

    Testing is an important part of the development process. If you have an expensive and complex application, do yourself a favor, and hire a test team. Don't try to do it yourself, unless you have no choice. You may do an OK job of it, but it sounds like you don't have the experience to do it properly. This isn't a critique of your skills, it is a fact that you have a different skillset. You don't expect an embedded programmer to develop web apps or vice-versa. Could they do it? Sure. But it will be a long and difficult road as they make the mistakes an experienced person already known how to avoid.

    If you have decided that comprehensive testing is important, (and some people decide it isn't), then do the right thing and get professional (test) help.

    • by Raenex (947668)
      While full-time testers are often made fun of, ("Those who can't code, test."), or it is considered a secondary role that developers can perform on their own, there is no replacement for an experienced tester.

      I've run across a few excellent QA people. They're extremely valuable if you can find them.

  • I've tried all of these but always end up getting sidetracked. Usually, the pain comes in terms of trying to apply tests to poorly designed or rapidly changing program models or in trying to set up test data in a db.

    Of these, I actually like SimpleTest the best so far, but I don't have more than a passing familiarity with any of them.

    In terms of code quality, I think I've got more mileage out of goodish coding practices

  • I'm a big fan of "Software Testing In The Real World: Improving The Process (ACM Press)", Edward Kit (ISBN 0201877562.) It got me thinking about testing at a time when I hadn't really gotten the "testing bug."
  • First off, you should be practicing TDD (Test Driven Development) as discribed here: http://en.wikipedia.org/wiki/Test_driven_developm e nt [wikipedia.org] as
    well as Continuous Integration as discribed here: http://www.martinfowler.com/articles/continuousInt egration.html [martinfowler.com].

    The next thing you need to do is hire some testers. TDD and CI can help make fantastic improvements in the quality of your code, but you will never replace the time and effort of another human working to break your code.
    • TDD - test driven development - means that you write tests before you write the code. Good TDD is actually BDD - behavior driven development. Write unit tests that are specifications for behavior. Continuous integration means that unit tests are automatically ran whenever code is integrated. Obviously this is just a matter of automation. Look for a test framework in the language you code in. There are lots of frameworks out there now. One way to whittle them down is to pick one that supports mock obj
  • The most common software testing solution that effectively tests a system of the complexity you have described is called "Production". Amazing how effective the "Production" environment is at locating all of the hard to find bugs. Anything less just won't do.

    Short of 'production', Mercury makes some excellent tools, but they're more for automated regression or load than for full coverage.
    • although I agree that a production environment can help make a good test effort it is not always practical or feasible and I think the poster actually needs to hear about best practices, methodologies, tools etc rather than just being told to get a production environment.

      Also, many unit and functional tests don't need a full blown production environment either -at least not until integration testing...

      If the production environment consists of $1 million worth of HW and dedicated outbound feeds to banks and
  • My take (Score:2, Interesting)

    by Pootie Tang (414915)
    Let me start by saying that there is no one size fits all
    solution. You described the nature of your projects to a degree, but
    not the nature of your employment.

    Do you work for a company or are you a contractor? Is your
    responsiblity just for your code or the application as a whole (which
    is another way of asking how big the team is in most cases)? Are the
    users of the software someone close to you, you can ask them "does this
    do what you want" or is this going to go on the shelf in a store and
    you have no idea wh
    • by rhartness (993048)
      Thanks for the response! Actually, I'm following many of your suggestions already. To give you (and others) a little background I work for a company that has one IT supervisor and I am the sole, dedicated software engineer. We are a manufacturing company and the point of the project that I am currently working on is to build an advanced, in-house system that will be used by hundreds of people all across the globe, some in very remote locations for internal tracking. The idea for the system was the brain
      • Re: (Score:2, Insightful)

        by jt2190 (645297)

        ...when requests come up (on an almost weekly bases) to change the code and rework specific modules entirely at the whim of management, things start to get complicated.

        Willy-nilly changes are a huge risk to your success. Your management needs to focus on a goal, and let you reach it. It's like running twenty-five miles of a marathon and then having them move the finish line one you: Pretty soon, you drop dead from exhaustion. You can help them focus by making a list of all the stuff that they don't keep

      • Re: (Score:3, Insightful)

        by Pootie Tang (414915)
        Given the additional information, I have a little more to say. It reinforces my impression that you are basically on the right track, there are just no silver bullets. However I think unit testing is probably more significant for you than I assumed at first. I think jt2190 expressed the value of unit tests very well.

        I also agree with jt2190 about the significance of requirement changes. This is absolutely the area where you stand to make the most gains. If your boss does not have professional software devel
  • Someone above recommended Edward Kit's book. Definitely worth reading, imo.

    But the biggest reason I see for developers not being so good at testing, is that they are of the mind set that they try to show how the SUT does work. Not really in the least bit helpful, that is what your unit tests should have been doing. No, the mind set is 'In what ways does this fail?'. All software fails, you just have to find out how. That is probably one of the reasons you think so much of the literature is of a philosoph
    • by Raenex (947668)

      But the biggest reason I see for developers not being so good at testing, is that they are of the mind set that they try to show how the SUT does work. Not really in the least bit helpful, that is what your unit tests should have been doing.

      I have no idea what "SUT" stands for, but testing basic functionality is also important at higher levels than units. Integration testing seems to be the common phrase, though Wikipedia also mentions systems testing (not that I can discern a difference).

  • by javaxman (705658) on Thursday November 30, 2006 @07:30PM (#17059024) Journal
    First, you should write a test plan. Actually, scratch that... first, you should write a design document. You have one of those, right? If not, you don't know what you're testing, and will eventually fail to cover some portion of your product, and that portion will likely break. Once you have a halfway decent design document, write a test plan that refers to your design document.

    I'm going to second the notion I've seen in other posts that if your project really is big, there should be a programmer or analyst dedicated to testing it. You can hire someone who is less of a programmer and more of a tester if you want, but you'll get the best coverage if you can find someone with real programming chops who is able to design and implement tests. If you're doing your project right, that QA team will be every bit as involved in every portion of your project as you are.

    Without knowing more about your particular project, it's very difficult to suggest appropriate tools. Different types of systems require different approaches to testing.
  • Verification? (Score:3, Insightful)

    by RAMMS+EIN (578166) on Thursday November 30, 2006 @08:32PM (#17059688) Homepage Journal
    As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took. I don't know if there are any verification tools for the programming language you are using (the ones I used were all for toy languages), but that was my first thought.
    • Re: (Score:1, Insightful)

      by ChefAndCoder (902506)
      Formal verification is one of those last vestiges (especially in academia) of the idea that it is possible to prove that software programs are correct. It's true for very small focused programs which do not have to interact with anything i.e. self contained. In practice, the effort required to formally prove that your software is correct is impractical. For "real" software systems, the number of unknowns introduced via API's, O/S and Hardware Interfaces, even just user interaction makes such "proofs" grow
    • Re: (Score:2, Insightful)

      by Raenex (947668)

      As a software engineer, I would have expected you to rely less on sanity tests, and more on formal verification. That's what we did in the (few) software engineering courses that I took.

      In the real world very little software is formally verified. It's very expensive and time-consuming, and the vast majority of programmers don't know how to do it. Most projects just cannot afford it.

      You're making too big a deal out of his "software engineer" label. There's no such thing as a software engineer, in

  • [url]http://portal.acm.org/citation.cfm?id=1138929 .1138952&coll=GUIDE&dl=&type=series&idx=1138929&pa rt=Proceedings&WantType=Proceedings&title=Internat ional%20Conference%20on%20Software%20Engineering&C FID=15151515&CFTOKEN=6184618[/url]
  • Ahem... try this (Score:3, Informative)

    by plopez (54068) on Thursday November 30, 2006 @11:10PM (#17061000) Journal
    Go to wikipedia, look up software testing. Click aon some of the referenced links, then chase those links and you will find a wealth of information.

    Just google +"software testing" you will find tons of links.

    you might try these guys: http://www.softwareqatest.com/ [softwareqatest.com]
    They have some nice links as well.

    Or would you rather start with a newsgroup? Go to google news groups and seach on software testing again. Lots of knowledgable and helpful folks hang out on those boards.

    BTW, a colorful phrase for "trying to test everything a user might do" is "soap opera testing". Google it, the story behind the name is sort of fun.
  • STAF [sourceforge.net] will support you in automating tests.
    The effort to implement something will approximately double, but you'll have a continuously regression tested system/project.
  • If you have the option: I do all of the testing that I can, but I also write in that in the event of an exception write to a log (standard) and then email to me an exception report. The exception report typically contains the user that hit it, the page/file that hit it, and method that returned the exception. In the morning I drink coffee and read fix up the exceptions. Luckily, I haven't received any lately (fingers crossed).

"Text processing has made it possible to right-justify any idea, even one which cannot be justified on any other grounds." -- J. Finnegan, USC.

Working...