Seeking Input for Software Verification Policies? 24
e8johan asks: "I currently work at a company in the automotive industry. It is importat to ensure quality and for our products we use extensive testing before, during and after the assembly. Now I have been asked to help in the creation of a company instuction for ensuring that all 'in-house' developed software works properly. I have written a short summary describing the essentials such as: "do not ignore compiler warnings", "always describe what the code is intended to do in your comments", "do not assume that all users run in the same environment", etc. I now feel that I need more input, has anyone done anything like this before? How do you test your software?"
Same stuff. (Score:1)
That said, it depends on what I'm doing as to whether the test cases are automated or a quick checklist.
Last, I've yet to figure out how to effectively test for bugs in the test case code
Unit tests & automation (Score:2, Informative)
Read all the extreme programming practices. No, you don't have to follow them (any or all). But you should READ them and WHY they were suggested. Knowing why will help you.
Standard code practices are good. Anything that can help you automate things is good.
My software can log every memory allocation and uses a quick n dirty perl script to ensure that each was deallocated (I write code with ZERO memory leaks and no garbage collection). Consider similar practices.
Work your ass off. There is no magic toggle switch somewhere you can flip between hell and perfection, you gotta do it yourself. But I do and it aint that hard. You can too.
Make the first lines in ALL of your procedures... (Score:3, Insightful)
Testing or proving? (Score:3, Interesting)
You need good things like loop invariants, and deep thought about what you wish to limit yourself to. But, if you start at the base procedures and work to building up to the ones that call these then you can work your way up to "if the input from user matches..." and then put in tons of good input checking and you're solid.
Now, this takes lots and lots of time, but maybe this is the sort of things you're looking to do. I used this method in designing an e-voting system to prove that things were encrypted properly and that the system would be stable.
Fun stuff.
Hand it to a Noob (Score:1)
Try high level (Score:1)
The same way I'll use ed for a quick jot or one line change, vi for a fiew quick changes or a letter, and fire up emacs (in vi emulation mode) if I'm gonna be restructuring and debugging my code for more than an hour straight, I think it makes sense to do a small program that needs to run as fast as possible in c, but a larger or more featureful program in something like lisp or python, where you sweat less of the small stuff.
NB s/first \(order\)/higher \1/ (Score:1)
switch "first order" with "higher order" in my previous comment for it to make sense.
Story title (Score:1)
With this disclaimer, the title of this story should be a statement rather than a question, unless the editor is unsure wheter the submitter was asking a question.
Obvious, but important (Score:1)
Depending on how critical it is that the stuff is correct, consider looking at least at the basis behind SPARK (High-Integrity Ada) http://www.sparkada.com/ too.
Yikes.. (Score:3, Insightful)
Sounds like you are not going far enough... lives are at stake if this is code for a car, and handing a bunch of C hackers some guidelines is not going to cut it. In general, I think that when you require absolute correctness, software systems are just too complex to try to tackle by brute mindpower. I really believe that you need to develop automated systems for checking that code is correct -- or instance, I understand that BMW does model checking on their code to make sure that bad things can't happen (ie, deadlock).
Some recommendations (Score:4, Insightful)
First you should understand the difference between testing and verification. With testing, you rule out a certain number of bugs, but you can never guarantee the absence of bugs. The latter is only possible with software verification, where you prove that the software contains no bugs and fulfills its specification.
Unfortunately, software verification is very expensive. Also, software verification is still a subject of research, and AFAIK it has not been tried for large software projects so far. (My current research project aims at verifying a complete operating-system kernel -- check out the VFiasco project [vfiasco.org].)
So in most cases, verification is too expensive, and you need to ensure robustness by other means like a good programming standard and testing. (I believe that is what you are looking for, despite the title of your question including the word ``verification.'')
If your shop uses C, I recommend the following book: Les Hatton, Safer C [amazon.com]. It explains what a good programming standard should contain, and advocates automatic enforcability. (It also explains what the standard should not contain -- in particular, issues of style such as indentation and variable naming.) One example of such a programming standard for C are the Guidelines for the Use of the C Language in Vehicle Based Software [misra.org.uk], which should be particularly relevant to you as you also work in the automotive industry.
Testing is another important aspect. In safety-related environments such as yours, a high test coverage for your code is desirable. This requirements suggests automated testing on the unit level. Automated unit testing is practical only if the code is designed to be testable, and there are a number of strategies which provide this property, such as avoiding dependency cycles (for C and C++, John Lakos' Large-Scale C++ Software Design [amazon.com] still is the ultimate guide), or creating the unit test before the program (the Extreme Programming [extremeprogramming.org] way). I also liked Testing Object-Oriented Systems: Models, Patterns, and Tools by Robert Binder [amazon.com] - a huge and thorough guide to all imaginable kinds of testing, especially for object-oriented systems.
Maybe a little obvious / easy (Score:4, Informative)
Here's a few random things that occur to me (most are probably repeats of other posts, though):
Automate as much as you can (testing, code generation, anything that's possible to automate - do it).
Use strongly typed languages, and strongly typed interfaces.
Use any and all warning flags, if available
Whenver you find a bug, add a test that detects that bug to your test suite. Then, and only then, fix the bug.
Read *this* book: The Pragmatic Programmer (Score:3, Insightful)
you might want to have a look at this book:
http://books.slashdot.org/article.pl?sid=00/02/15
http://www.pragmaticprogrammer.com/
"The Pragmatic Programmer". They have a chapter that outlines the general principles of testing your software... And all the other chapters are very well worth their money!
Comments considered harmful (Score:4, Insightful)
This can hinder the finding of bugs. Too often I have seen people read the comments, say "oh, that's what it does" and go on without actually looking at the code.
A much better idea: write clear code, with good variable names, well abstracted and well named procedures / functions / methods, so that the code itself is the "documentation."
Save the comments for out-of-band information like citations, etc.
-- MarkusQ
Re:Comments considered harmful (Score:1)
Seperate test from development (Score:3, Insightful)
Project leads must make sure that specifications are made before coding starts.
Test should primarily (but not entirely) be concerned with how the systems should work -- and they need to verify that the specifications are complete and usable. This means bringing test (tester/VV&T) on before coding starts.
Developers must develop to those specifications.
If something changes, the PM should OK a specification change. No unilatteral code changes. Keep in mind that this does not eliminate creativity, it does mean that if some new feature is added it gets doccumented and tested. A bad PM will not be flexible on this point. A bad developer will ignore the process. One can lead to the other.
Initially, developers will not like it -- and some parts they will never like (the extra paper work, someone finding stupid errors constantly...). In the end, few will hate the idea and most will like the benifits...even if it is annoying at times.
Can there be exceptions to any of this? There will be -- but keep the exceptions rare. A good ballence is important.
Documentation, test harnesses (Score:1)
Something I have actually been taught in college. Good documentation is very important. Everything should be documented well enought that another programmer will be able to sit down and understand the code clearly. Pictures also help a lot too.
For testing have something that will test each function/method in the code. So that if you have f(4) = 5 run a test that makes sure this function returns that.
Also try reading about egoless programming. I read a little about it and found it to be very interesting.
Thank you Felleisen
Lead programer is boss, and code review team (Score:2)
first of all, make sure your tech lead is boss, and has the power to force his vision on everyone. When the tech lead says "Do it this way", then it is done that way. Of course that means your tech lead is good. (Often call cheif architect) He is not a manager, though the buck stops with him for the schedule. He needs to be a good programer who understands every part of the code, and can change any of it. (and have it work). He will not have time to touch every part of the code, but if he can't understand it he will make you re-write it.
Second, have a code review team. A small (3-6?) group of people who do nothing be review code all day every day. They know the code better than the chief architect, but all their job is, is to make sure it is bug free, well styled (readable), and matches the spec. They answer to the tech lead.
The chief architect is critical, I don't think any project can succede without a good dictator for a tech lead. The code reviewers are a good idea for a large project, for a small project they are the coders themselves. (for a large project you cannot understand every part of the code and get your work done, while for a small project it is easier to get your work done when you understand it all)
Management is going to be the biggest factor in success.
More than testing (Score:1)