Seeking Input for Software Verification Policies?

e8johan asks: "I currently work at a company in the automotive industry. It is importat to ensure quality and for our products we use extensive testing before, during and after the assembly. Now I have been asked to help in the creation of a company instuction for ensuring that all 'in-house' developed software works properly. I have written a short summary describing the essentials such as: "do not ignore compiler warnings", "always describe what the code is intended to do in your comments", "do not assume that all users run in the same environment", etc. I now feel that I need more input, has anyone done anything like this before? How do you test your software?"

Same stuff.

[rtaylor]

Layout your guidelines of what you expect the software to handle before it's written. Write or create test cases which will check for the guidelines. Now write the software and fix it until it passes all of the tests.

That said, it depends on what I'm doing as to whether the test cases are automated or a quick checklist.

Last, I've yet to figure out how to effectively test for bugs in the test case code ;)

Unit tests & automation

[Pyromage]

Use automated unit tests. It is true that it won't catch everything, but nothing will. The only system operating perfectly is off. Unit testing aint a panacea of perfection, but its a heck of a lot closer than "Well, the numbers /looked/ right!".

Read all the extreme programming practices. No, you don't have to follow them (any or all). But you should READ them and WHY they were suggested. Knowing why will help you.

Standard code practices are good. Anything that can help you automate things is good.

My software can log every memory allocation and uses a quick n dirty perl script to ensure that each was deallocated (I write code with ZERO memory leaks and no garbage collection). Consider similar practices.

Work your ass off. There is no magic toggle switch somewhere you can flip between hell and perfection, you gotta do it yourself. But I do and it aint that hard. You can too.

Make the first lines in ALL of your procedures...

[hackwrench]

code to check for bounds. Too many programs don't, just figuring that the values of a byte, integer, long int are sufficient, and leaving it up to other procedures or libraries to check.

Testing or proving?

[heliocentric]

There is a system where if you set your pre and post conditions well you can mathematically prove your code performs what is asked. You "simply" drag your post conditions up your code and if you arrive at a set where the pre conditions are at least a sub set, then you know if the input matches the post conditions will also be satisfied.

You need good things like loop invariants, and deep thought about what you wish to limit yourself to. But, if you start at the base procedures and work to building up to the ones that call these then you can work your way up to "if the input from user matches..." and then put in tons of good input checking and you're solid.

Now, this takes lots and lots of time, but maybe this is the sort of things you're looking to do. I used this method in designing an e-voting system to prove that things were encrypted properly and that the system would be stable.

Fun stuff.

Hand it to a Noob

[BagOBones]

In a test environment hand the app to someone who only had a basic understanding of how to the system works and let them try data entry.. They will often find many holes in your validation. What happens if someone leaves a field blank? Enters a zero or types a character in a number field.. Does it post and error or does it crash and burn? What happens when a user presses buttons out of order or closes a screen before completing a form?

Try high level

[caca_phony]

In general, the higher the level at which you can keep your work, the faster you will churn it out and the fewer errors. This can involve everything from reuse of proven snippets to well written libraries to very high level languages. A note about very high level languages (lisp, ML, Haskell, and pals) is that you don't even know how usefull their features are until you learn the language and find them missing somewhere else. It was a revelation to me about programming when I first wanted a first order function in C (and couldn't have it without a kludge). The only problem is that, as we all know, features and efficiency are having your cake and eating it.
The same way I'll use ed for a quick jot or one line change, vi for a fiew quick changes or a letter, and fire up emacs (in vi emulation mode) if I'm gonna be restructuring and debugging my code for more than an hour straight, I think it makes sense to do a small program that needs to run as fast as possible in c, but a larger or more featureful program in something like lisp or python, where you sweat less of the small stuff.

NB s/first \(order\)/higher \1/

[caca_phony]

oopsy
switch "first order" with "higher order" in my previous comment for it to make sense.

Story title

[caca_phony]

Though I understand that nitpicks are annoying, I find that rampant imprecision in language makes it hard for me to state clearly something unexpected without it being heard as what is already taken for granted.

With this disclaimer, the title of this story should be a statement rather than a question, unless the editor is unsure wheter the submitter was asking a question.

Obvious, but important

[headshrinker]

Define all the data types going in and out of all functions and what its purpose is. At least do this at the library level (any externally-accessible calls) and to be safe(r) internally too. Define all units too - it might seem obvious at the time, but not everyone works in Imperial, for example *cough*Mars Polar Lander*cough*.

Depending on how critical it is that the stuff is correct, consider looking at least at the basis behind SPARK (High-Integrity Ada) http://www.sparkada.com/ too.

Yikes..

[Tom7]

Sounds like you are not going far enough... lives are at stake if this is code for a car, and handing a bunch of C hackers some guidelines is not going to cut it. In general, I think that when you require absolute correctness, software systems are just too complex to try to tackle by brute mindpower. I really believe that you need to develop automated systems for checking that code is correct -- or instance, I understand that BMW does model checking on their code to make sure that bad things can't happen (ie, deadlock).

Some recommendations

[Read The Fine Manual]

(I work in an academic context, so take the following with a grain of salt.)

First you should understand the difference between testing and verification. With testing, you rule out a certain number of bugs, but you can never guarantee the absence of bugs. The latter is only possible with software verification, where you prove that the software contains no bugs and fulfills its specification.

Unfortunately, software verification is very expensive. Also, software verification is still a subject of research, and AFAIK it has not been tried for large software projects so far. (My current research project aims at verifying a complete operating-system kernel -- check out the VFiasco project [vfiasco.org].)

So in most cases, verification is too expensive, and you need to ensure robustness by other means like a good programming standard and testing. (I believe that is what you are looking for, despite the title of your question including the word ``verification.'')

If your shop uses C, I recommend the following book: Les Hatton, Safer C [amazon.com]. It explains what a good programming standard should contain, and advocates automatic enforcability. (It also explains what the standard should not contain -- in particular, issues of style such as indentation and variable naming.) One example of such a programming standard for C are the Guidelines for the Use of the C Language in Vehicle Based Software [misra.org.uk], which should be particularly relevant to you as you also work in the automotive industry.

Testing is another important aspect. In safety-related environments such as yours, a high test coverage for your code is desirable. This requirements suggests automated testing on the unit level. Automated unit testing is practical only if the code is designed to be testable, and there are a number of strategies which provide this property, such as avoiding dependency cycles (for C and C++, John Lakos' Large-Scale C++ Software Design [amazon.com] still is the ultimate guide), or creating the unit test before the program (the Extreme Programming [extremeprogramming.org] way). I also liked Testing Object-Oriented Systems: Models, Patterns, and Tools by Robert Binder [amazon.com] - a huge and thorough guide to all imaginable kinds of testing, especially for object-oriented systems.

Maybe a little obvious / easy

[randombit]

But if you haven't already, read the Practice of Programming. It has some good thoughts about testing applications that should help you out. And it's a pretty decent book, anyway.

Here's a few random things that occur to me (most are probably repeats of other posts, though):

Automate as much as you can (testing, code generation, anything that's possible to automate - do it).

Use strongly typed languages, and strongly typed interfaces.

Use any and all warning flags, if available

Whenver you find a bug, add a test that detects that bug to your test suite. Then, and only then, fix the bug.

Read *this* book: The Pragmatic Programmer

[cnvogel]

Hi,

you might want to have a look at this book:

http://books.slashdot.org/article.pl?sid=00/02/15/ 1912222

http://www.pragmaticprogrammer.com/

"The Pragmatic Programmer". They have a chapter that outlines the general principles of testing your software... And all the other chapters are very well worth their money!

Comments considered harmful

[MarkusQ]

"always describe what the code is intended to do in your comments",

This can hinder the finding of bugs. Too often I have seen people read the comments, say "oh, that's what it does" and go on without actually looking at the code.

A much better idea: write clear code, with good variable names, well abstracted and well named procedures / functions / methods, so that the code itself is the "documentation."

Save the comments for out-of-band information like citations, etc.

-- MarkusQ

Re:Comments considered harmful

[Hecatonchires]

I take pride in the fact that almost anyone can read my code and see what it does. I'm not really a very good programmer, but the only comments I ever use are function headers. Noone ever complains when its time to debug my code.

Seperate test from development

[Spoing]

Project leads must make sure that specifications are made before coding starts.

Test should primarily (but not entirely) be concerned with how the systems should work -- and they need to verify that the specifications are complete and usable. This means bringing test (tester/VV&T) on before coding starts.

Developers must develop to those specifications.

If something changes, the PM should OK a specification change. No unilatteral code changes. Keep in mind that this does not eliminate creativity, it does mean that if some new feature is added it gets doccumented and tested. A bad PM will not be flexible on this point. A bad developer will ignore the process. One can lead to the other.

Initially, developers will not like it -- and some parts they will never like (the extra paper work, someone finding stupid errors constantly...). In the end, few will hate the idea and most will like the benifits...even if it is annoying at times.

Can there be exceptions to any of this? There will be -- but keep the exceptions rare. A good ballence is important.

Documentation, test harnesses

[f0rtytw0]

and egoless programming
Something I have actually been taught in college. Good documentation is very important. Everything should be documented well enought that another programmer will be able to sit down and understand the code clearly. Pictures also help a lot too.
For testing have something that will test each function/method in the code. So that if you have f(4) = 5 run a test that makes sure this function returns that.
Also try reading about egoless programming. I read a little about it and found it to be very interesting.
Thank you Felleisen

Lead programer is boss, and code review team

[bluGill]

first of all, make sure your tech lead is boss, and has the power to force his vision on everyone. When the tech lead says "Do it this way", then it is done that way. Of course that means your tech lead is good. (Often call cheif architect) He is not a manager, though the buck stops with him for the schedule. He needs to be a good programer who understands every part of the code, and can change any of it. (and have it work). He will not have time to touch every part of the code, but if he can't understand it he will make you re-write it.

Second, have a code review team. A small (3-6?) group of people who do nothing be review code all day every day. They know the code better than the chief architect, but all their job is, is to make sure it is bug free, well styled (readable), and matches the spec. They answer to the tech lead.

The chief architect is critical, I don't think any project can succede without a good dictator for a tech lead. The code reviewers are a good idea for a large project, for a small project they are the coders themselves. (for a large project you cannot understand every part of the code and get your work done, while for a small project it is easier to get your work done when you understand it all)

Management is going to be the biggest factor in success.

More than testing

[HeyLaughingBoy]

From your request, it appears that you are looking for information on more than just testing. If you are serious about quality, then the entire development process, from requirements to maintenance should be carefully considered. I assume your software products are not safety-critical or you wouldn't be asking the question here, but the auto industry probably already has standard frameworks for that kind of thing. If you have contacts working on automotive embedded development, asking them might be a good start. As far as books go, try Solid Software for process-oriented advice, or Writing Solid Code which is more coding-oriented, but also covers testing and requirements.