Tools for Debugging Stack Corruption? 30
blackcoot asks: "I know that there are tools which exist on hardened Linux distros and OpenBSD (and probably $your_os_of_choice too), which are designed to help track down stack corruption (which is often symptomatic of buffer overruns). Unfortunately, that's about all I know about those tools. What options are there? How effective are they? What does it take to get access to those tools? Are they really useful enough to make the effort justified?"
"My goal here is to increase my effectiveness at hunting down memory bugs, not necessarily to produce bullet proof, secure production quality code — the bugs I'm dealing with are, I believe, largely in software delivered by a subcontractor who swears they test their code and can't reproduce my bugs. What I really want is to a) demonstrate to them that a problem does, in fact, exist; b) demonstrate that the problem exists inside their code; and c) give them the tools they need to find, repair, and verify that the bug is no longer an issue.
First prize in my mind would be a Valgrind like tool which only requires trivial changes to the build process, but I'm pretty open to suggestions. If I have to run a hardened Linux to make this all possible, suggestions on pretty leading edge distros with reasonable automagic self configuration and hardware detection + laptop support would be greatly appreciated. Thanks much!"
Electric Fence (Score:4, Informative)
Boehm Garbage Collector (Score:3, Informative)
A Couple Of Tools (Score:5, Informative)
First of all, there's libsafe [avayalabs.com] which is just a simple compile and install. Seeing as we don't know much about your specific problem, I'm not sure if it'll help.
Second to try is compiling your own custom kernel with the GrSecurity [grsecurity.net] kernel patch. It has as part of it the PaX [grsecurity.net] kernel patch which is very effective at protecting against overflows. You could even just install the PaX kernel patch itself, but I believe the version in GrSecurity is kept more up-to-date. You can compile it with protection turned off by default, but using the PaX tools turn on protection for just the binaries you wish to check.
Installing either (or both) of these could well help you, without the need to blow away your current install and start fresh.
There are obviously several alternatives. (Score:5, Informative)
Using a complete hardened Linux distro is not necessary for "normal" development work, but it may be a good idea to verify that your application actually works there too.
In addition to the run-time checkers you can also look for static checkers like Splint [splint.org]. It can provide you with some extra information about potential problems that only occurs under certain conditions that maybe aren't met during runtime.
If the effort of trying to track down stack corruption is worth it? - YES! Absolutely! Catching bugs in an early stage is essential to keep down the lifecycle cost of a system. Also consider the risk of badwill if your product is prone to strange behavior and crashes.
Re:A Couple Of Tools (Score:4, Informative)
Re:There are obviously several alternatives. (Score:3, Informative)
ElectricFence, valgrind, Boehm GC, Purify, etc are all heap debuggers (for finding problems with overruning malloc'd memory, memory leaks, etc). It's possible that Purify has stack debugging capability these days, I'm not sure.
CCured (Score:3, Informative)
All good suggestions, but not for stack (Score:5, Informative)
I feel your pain about bugs in libraries that you must use without the source code. I had an arrangement like that for nearly two years with extremely buggy code. Just relinking the static library with changes to my code, changing where in memory the library would reside, would often cause huge problems. Let's just say I got really good at debugging in assembly with gdb. I got where I could call them up and say something like, "you have some code at the end of function foo that looks like 'a[2] = b', but a was never allocated." They'd always reply with something like, "Yes it is ... oh wait..."