Making the Case for Better Bugtracking Tools?

WeNeedBetterTools asks: "Help! I am trying to convince the leaders of my development organization that the quality of our bugtracking tool has a meaningful impact on the quality of products we produce. Right now, the tool we have is difficult to use because there is a high overhead to creating bugs, and they are very difficult to find in the system once they're reported. My main argument is that the easier the tool is to use, the more people will be motivated to enter bugs. My managers argue that if people aren't entering bugs, it's their own damn problem, and we ought to crack the whip until they always file every bug they possibly can. I need some solid references for my position. JoelOnSoftware has a great argument for making it simple to enter bugs, but he lacks credibility with my managers by virtue of the fact that he is also selling a bugtracking tool. Can anyone point me to any solid research on the relationship between the quality of bugtracking tool and the impact on product quality?"

so true

[oever]

there is a high overhead to creating bugs

I can confirm this. I introduced some bugs in my code today, but boy, am I tired right now. But hey, every bug counts.

Been there...

[walt-sjc]

The problem is larger than just the ease of use of the tool. The more people you have entering bugs the more they are going to get categorized wrong, and the more dupes you are going to have, etc.

Assign someone in QA as the "bugmaster". Possibly rotate the responsability between team members so you don't get burnout.

Have people email bug reports to the bugmaster and have that person enter the bugs into the system the correct way. Yeah, it can be a lot of work for the bugmaster depending on the size of the system, but the QUALITY of the data will be vastly improved.

Trust me on this. A tool will not fix a process problem, no matter HOW easy a tool is to use. If you ARE choosing a new tool, make sure that ANY tool you use is web-browser usable for ALL functionality (avoid IE only, active-x required tools)

Re:Been there...

[SarcasticTester]

Sounds to me as a very very expensive way. THe way I always work is simply using Bugzilla (make it the homepage on all machines for all I care!) and have the bugmaster run through all issues at least once a day, he/she can than decide who's gonna get the bugs in order to fix em and also check for dupes. Bugzilla is by far easiest and cheapest to implement in any environment. It is free and webbased and moreover, it is very easy to customize it. If your management doesn't want to spend too much money on quality it is their problem, but keeping your work fun and somewhat easy is your own. So just implement a solution and after a couple of weeks show the improved results to management. Another thing to improve stuff is by implementing UnitTesting. Have developers write UnitTests before writing any code and run your unittests with every build (Contiuous Integration is Heaven for testers!!) If Unit tests fail, it is no use testing software by hand and you can start kicking developers around.

Re:Been there...

[dubl-u]

A tool will not fix a process problem, no matter HOW easy a tool is to use.

Hmmm... My experience is different.

A good working environment is designed to make good things easy and bad things hard. For example, I write a lot more unit tests when I have a good unit testing framework. I run them a lot more when it's part of my build process. And I do a lot more refactoring when I use a tool that takes care of the boring parts of it.

But I'd agree that no tools can substitute for a responsible human. A bugmaster should be a big help in introducing the tool, especially if the bugmaster is somebody respected by all.

But better still is the team coming together and deciding, "Damnit, we need to track bugs better! We want an easier tool!" Anything imposed above from the managers, especially ones who say things as painfully dumb as, "we ought to crack the whip" is much less likely to succeed than something adopted by consensus.

Show some initiative

[Col. Klink (retired)]

Bugzilla's free (libre). Just install it. Managers always think everything's going to take a lot of time and they never foresee the benefits. On a Debian system, "apt-get bugzilla" and your done.

Most new technologies, Linux included, are installed in the enterprise by real people doing real work. Real Engineers(tm) don't wait around for the PHBs to realize a trend that's been going on for 6 years.

A Good Boss(tm) is one who sees that things work better now than before, and nothing more. If you expect more from a boss, you're delusional and you may as well start applying for management positions yourself.

And always remember: it's easier to get forgiveness than approval.

Re:Show some initiative

[Clover_Kicker]

>Bugzilla's free (libre). Just install it.

Um, what about the rest of the team? How's he going to get the rest of the developers to use Bugzilla if the managers aren't on his side?

Re:Show some initiative

[Col. Klink (retired)]

I think the developers will be the first to jump on the bandwagon. Just give them the bugzilla URL and show them how easy it is to add and track bugs. You don't even have to tell management until it's well entrenched.

Re:Show some initiative

[Clover_Kicker]

>I think the developers will be the first to jump on the bandwagon. Just
>give them the bugzilla URL and show them how easy it is to add and
>track bugs. You don't even have to tell management until it's well
>entrenched.

You must work with smarter and/or nicer people then I do :)

Re:Show some initiative

[duffbeer703]

I've been considering installing Bugzilla and find it to be a real pain in the neck to get running.

If you are not using Debian, the installation process is complicated, cumbersome, and time consuming.

Hopefully someday Bugzilla will run on something other than Linux/MySql.

Re:Show some initiative

[Electrum]

I agree. Bugzilla is overkill for most people. More features doesn't mean better product.

I tried out a lot of bug trackers, and I like Mantis [sourceforge.net] the best. It is simple, yet has the features needed to get the job done. It doesn't take hours to setup like Bugzilla. The user interface is much simpler, so filing bugs and using the system isn't a huge chore.

Re:Show some initiative

[babbage]

No, please don't use Bugzilla -- it's reputation far exceeds its actual quality. Bugzilla is an arcane, tightly bundled colledtion of hard to extend CGI scripts sitting on top of a bizarre MySQL schema. If it doesn't exactly meet your needs (i.e. you are not the Mozilla project), extending it can be a nightmare.

May I humbly suggest that you take a look at RT: Request Tracker [bestpractical.com] instead. RT is a general purpose ticketing system [bestpractical.com], suitable not only for bug tracking, for for all kinds of organized message exchange within an organization (i.e. help desk, sales force tracking, some aspects of inventory management, etc). RT allows users to collaborate via a web interface, email, or the command line. By providing multiple interaction interfaces, RT encourages users to work with the system by communicating the way they would already, rather than working against them by forcing them to adapt to a wholly new system. If you don't like the web interface, feel free to change it. If it's still not enough, people can just use email instead -- just cc: your RT account on ticket related mails, and include the ticket number in the subject line. Hey presto, people can do almost what they were doing in the first place.

RT is written in clean, OO Perl [bestpractical.com] making wise use of CPAN libraries instead of implementing everything from scratch. It will run on a variety of operating systems & databases (MySQL, PostgreSQL, Oracle). The system is well documented [fsck.com], easily extensible [fsck.com], and comes with a vibrant & supportive user community [bestpractical.com]. It can even be integrated with things like pagers, so that the creation of critical tickets can send out a pager message to key personnel.

All in all, RT is a very nice, very well engineered system that IMO is far more suitable for most users than Bugzilla, for which the suitable scope is much more restricted. That's why RT is now being used in, among other places, Perl's bug tracker [bestpractical.com] at rt.perl.org [perl.org].

Disclaimer: My company uses RT, and I have met Jesse Vincent, RT maintainer, a handful of times, and even though I think it would be pretty cool if people switched to RT and bought support contracts [bestpractical.com] from Jesse, I have nothing to gain if any of this happens. I just sincerely think that RT is better software than Bugzilla for almost all users, and would like to see development of the software continue to flourish and become accepted more widely. Spend a week messing around with RT and IMO you'll never want to go back to Bugzilla...

not enough info

[josephgrossberg]

Why don't your managers want to try something new? If it's price, tell them bugzilla is free, and is used by companies from Red Hat [redhat.com] to Netscape [mozilla.com]. If it's a political issue, that's a whole 'nother story.

Also, why is there so much overhead? Entering a bug should be as simple as filling out a single HTML form. Is your UI too big and complicated? You can probably strip it down to the essentials, without discarding it entirely. Is it on more than one page? You can probably condense them into one form. Are the bugs stored in a database? You can write your own search page.

And if you said which bug software you're currently using, you could probably have gotten more specific and relevant tips on how to make the best use of it (change the configuration, prepopulate certain fields, roll your own modified templates, etc.).

Just do it

[SarcasticTester]

THe way I always work is simply using Bugzilla (make it the homepage on all machines for all I care!) and have the bugmaster run through all issues at least once a day, he/she can than decide who's gonna get the bugs in order to fix em and also check for dupes.

Bugzilla is by far easiest and cheapest to implement in any environment. It is free and webbased and moreover, it is very easy to customize it. If your management doesn't want to spend too much money on quality it is their problem, but keeping your work fun and somewhat easy is your own.

So just implement a solution and after a couple of weeks show the improved results to management.

Another thing to improve stuff is by implementing UnitTesting. Have developers write UnitTests before writing any code and run your unittests with every build (Contiuous Integration is Heaven for testers!!)
If Unit tests fail, it is no use testing software by hand and you can start kicking developers around.

Re:Just do it

[DarkKnightRadick]

In particular, look at how Bugzilla has helped the Mozilla project.

Re:Just do it

[dubl-u]

Have developers write UnitTests

Amen! Any development shop should have a box that just runs 24x7 checking out the latest code, doing a build, and running all the automated unit tests (written by the programmers) and functional tests (written by QA). Whenever a test fails, the responsible developer should be immediately e-mailed.

Then if you get your developers in the habit of doing test-first development, which gets you circa 100% coverage, your bug counts will drop significantly. I worked intermittently on a high-traffic dynamic website that has had a total of zero downtime and two or three shipped bugs, all minor, after about six man-months of development and nine months in deployment.

Re:Just do it

[scrytch]

> Bugzilla is by far easiest

I have yet to see a single bugzilla that even came close to end-user ease of use of Request Tracker. [bestpractical.com]. Bugzilla might have a million knobs and dials to twist, but that's precisely what I don't want to expose submitters to.

No research, but experience

[slappy]

I doubt you'll find much hard research to support your position, but you'll no doubt find a lot of anecdotal evidence. It seems pretty clear to me that the simpler it is to enter and track bugs, the more use a bug database will get, and I've seen that happen a number of times, especially with less technical users who simply aren't inclined to become masters of a bug tracking package. This is why I wrote my own free substitute for bugzilla [sourceforge.net] (shameless plug). This is certainly one software category that hasn't produce a globally ideal solution yet.

Re:No research, but experience

[sohp]

Hard to say what will motivate a manager, but in the case of the folks using the tools on a regular basis, it can be effective to talk to them about the way you imagine bug tracking should work, the benefits they get from those practices, and then talk about how your current tool supports or inhibits those practices.

The first two steps, practices and their benefits, are something I've done a little research on, and put together a set of Defect Tracking Patterns [advogato.org] that discuss how to work and the benefits of those practices in context.

Maybe if you can convince enough of the folks to want to adopt good practices, and that the current tool does not work well for those, you'll have some leverage with THe Boss.

Fight hard

[image]

Hi. I am the senior software architect for a rather large and well-known company doing business on the internet (~$1M/day in online revenue). One of the first things I did after taking this position, with the goal of turning the development team (100+ engineers) into a top-flight software house, was put a process in place to use a bug-tracking system not just for post-deployment defect control, but also during the development phase. Each task is tracked through a ticket and tied to a particular branch in our source control system. This creates a running log of not just the work itself, but also the thought that goes into the work.

While there was some resistance at first among developers and managers that had never done this before, over a few short months our code quality has improved and our production increased. In fact, the final case was made when comparing two teams of roughly equal size, one using the tracking system during development, one not. The one that was using it not only had fewer defects, but had written about twice as much code in the same time period. And the real benefit will come a year or two from now when we need to go back and triage a decision or issue long after the original engineers have moved around.

We happen to be using bugzilla and cvs as they are free, easy to modify to meet our needs, and familiar, but you could do this with just about any tool that allows you to track arbitrary issues.

Good luck -- and don't give up.

Corrective Action Process

[bwt]

The bug tracking system is almost never the problem with an organization's quality improvement effort. Never-the-less time spent entering and retrieving information into the bug tracking system is 100% non-value added waste. It should be minimized by using good technology if it can be done cheaply. However, in no case does the software solve problems for you or prevent you from having to get your hands dirty. If your organization isn't eyeballing each and every problem, assigning ownership, prioritizing, and tracking improvements to closure, you are just going to fail more with a better system.

Bugzilla is cheap (free) and good, but even more importantly, you can see how other groups build a cradle to grave corrective action process around it. My advice to you is to give your boss a tour of the Mozilla bug tracking **process** with their copy of bugzilla as your window. Set up bugzilla on your own machine and play-test it by re-entering a few of your own bugs into it and mock-working them.

use talkback

[scrytch]

god only knows where you can get the talkback quality feedback agent (fullcircle.com is a different company now) but you can put something similar into your app. Build for your internal dev folks an extra info field in so that they can add notes to the reports. Add a "report bug" menu option to the app, and make it only require a very few options to fill in (system affected, severity, summary, and notes). Give out a prize for the person who reports the most bugs (we are assuming professionals are not going to spam you with bogus reports). Pay for the prize yourself if you have to (it could be a big candy bar or $10 gift cert for starters) then get management to pay for it once you get things going.

Don't kvetch endlessly about specifics in the interface, just say "it needs to be easy and fast to report bugs. average time to file a bug report (now): xxx minutes. goal: xxx seconds."

Re:use talkback

[kawika]

Darn, I'm not a mod right now or you'd be +1 richer.

If you make it easy for users to file bug reports they'll do a lot of the work for you. Yes, someone needs to remove duplicates and separate out the feature requests but the bugs that really count are the ones that your users are seeing.

If you're looking to exterminate crash-type bugs, you might check into BugToaster [bugtoaster.com]. The basic data is free for the taking on their site, but detailed reports will cost you.

bake the error capturing right into the app

[avi33]

Why not invest some time and effort into effectively capturing system state at the time of the error? Your users won't complain so much if they are presented with a form page that already did half the work for them. Of course, it would be helpful for them to describe the circumstances as well, but you wouldn't be patching holes that don't exist because they're lazy when describing the issue.

While this is no silver bullet, if it's a web-based app, it should be trivial to capture the variables and pass them into your form. If it's not an error than can be caught by the system, how about a function on every "page" in your app that just grabs the relevant variables, making it easy for users to cut and paste. It will be easy to filter them if they are all somewhat standardized.

Pitfalls of The System

[4of12]

No one disputes that tracking bugs and fixing them isn't an important thing to do.

Where there is contention and differing points of view is when it comes to specific bug tracking tools.

Many folks, like me, resent heavy, overly bureaucratic tools. You know the kind: everyone has "roles" and there are authorized bug managers, assistant deputy bug fixers, project manager, etc., and every single keystroke about fixing the bug in the code requires 10 corresponding keystrokes of entry into The System.

The System sells well to managers because they get shown screen shots displaying exactly who is doing what, when, and how the bugs are being managed so incredibly well. Wow! What detail! How impressive!

The managers buying the system don't actually use it, though. Their staff do. And the staff suffers and learns to avoid The System at all costs so they can get some real work done - like fixing bugs for example.

Make sure that you evaluate potential bug tracking systems with the actual people who will be using it.

Other plusses to look for are a simple linear learning curve. Avoid the ones where you have to upload your brain with 4 days of training classes in order to use The System.

Tell the managers

[hackwrench]

that if people aren't buying the software because it's buggy because developers didn't report bugs, it's going to be their problem when they have to take a pay cut or look for another job.

Re:Tell the managers

[toast0]

Depending on the market, the buggyness has no effect on sales.

For instance, most consumers aren't worried about buggyness, as long as its not terribly buggy.

However, aerospace kinda likes bugless code.

more details

[Anonymous Coward]

As the submitter of this story, I want to answer some questions that have been raised. I am posting anonymously because I could get into some deep trouble for posting our internal woes in a public forum.

In no particular order:

The system in use today is an internally developed tool. It is a web form, but it so completely fubar that most of the constructive suggestions about automated submission do not apply. We have tried. Thanks for sharing your ideas.

To expand on the fubar-ness of the tool -- as you fill out the form to create a bug, several fields will submit a request back to the server each time you change their value. There are at least 8 such fields on the entry form. Each roundtrip lasts ~3-5 seconds. When you save a bug, you're waiting ~15-20 seconds for it to save.

the quality of bugs entered is a problem, but not the primary problem. A bigger issue is that bugs don't capture information that people have about the bug, because opening, editing, and saving a bug is a major hassle. Thanks for input on this matter, but I feel it's mostly under control.

Bugzilla... I only wish we could be lucky enough to use it! When arguing to my managers for bugzilla, I have found huge resistance to the idea of using an open source tool. Which is disappointing, because we use plenty of other such tools internally (ant, JUnit, log4j, xerces, etc, etc)

putting up my own tool seems to be of limited benefit. There are too many people entering bugs (QA, developers, customer support) to generate a groundswell of support for anything other than the tool in use today.

testing is not a major problem. We have a great set of tests to cover almost every bug we've encountered.

What I'm trying to do here is convince someone that people need good tools to do good work. I expect that to be a truism, but it is not being accepted as such. Does anyone know of any kind of study showing a change in bugs filed following the introduction of an easier-to-use bugtracking system?

Re:more details

[psycho_tinman]

Just a suggestion, based on similar experience with bug tracking software and getting management approval..

Part of the problem might be that its open-source software, part of the problem might be that it (if you consider Bugzilla) written in Perl, a language that your managment may not be comfortable with..

I'd suggest you test the waters with Scarab [tigris.org], implemented using servlets.. that might go down better..since you obviously seem to develop using Java (based on the other open source tools you cited)

If you are sure the choice of language isnt the issue, I would HIGHLY recommend Mantis [sourceforge.net], a simple, VERY useful bug tracker written in PHP.. I use this myself, and its very small and fast to configure (only problem I have with it right now is that it can only use MySQL as a source database)..

Sometimes, the easiest way to get bug tracking software in a large org. is to install it and start using it personally (ie: for bugs related to modules that you develop).. People will gradually come to appreciate exactly how much easier it is on your system as opposed to the other thing, and there will be a creep towards an easier to use system that cant be ignored..

Sometimes, the persons in charge of decisions like this simply dont know how much easier bug tracking software can be than your present software, which is what holds them back :D

Hope this helps and good luck

Try looking at Roundup?

[Pingster]

I firmly agree with you that the obstacles to entering and communicating about bugs have to be kept as low as possible.

When i started working as a developer at ILM [ilm.com], there wasn't really any bug-tracking system in use. So i threw together a quick hack, on a weekend when i finally couldn't stand it any more (isn't that how so many software projects get started?).

Over the two years i was there, it grew, but it stuck to three core ideals:

1. Don't force the person entering the bug to enter anything other than a description. Just allow people to go back later and add details, change categories or priority settings if they want to.
2. Make it really easy for other developers to enter the conversation around the bug, so the discussion and activity surrounding the bug can be recorded with the bug.
3. Optimize the user interface to display the most useful information in the available screen space.

To meet the first two goals, the system was based on e-mail: anyone could just send a free-form e-mail message to the roundup address, and a new bug would be created; then anyone who replied to messages about the bug would be automatically added to a mini-mailing-list. Every bug got its own automatically managed mini-mailing-list.

Most other bug trackers are really bad at the third goal: they take up all kinds of space with management details that people don't really care about. What a developer really wants to see is the descriptions of the bugs, so Roundup maximizes the screen space for that.

It grew far beyond my original plans, in terms of the number of users and bugs logged. The back-end implementation was terrible (no database, just lots of little text files; performance was awful but it did have the redeeming factor that you could just use "tar" to archive them).

As far as i know, ILM R&D is still using it today, and they're running multiple instances of it to support different applications and teams.

You can get the source code and see some screenshots on a page about Roundup [zesty.ca], though i'm not developing it any more. A new group of people has picked up the torch and carried it on in a Sourceforge project [sf.net] that is alive and well. Their project is a complete rewrite of Roundup, originally based on a design document I wrote, but now much extended. I encourage you to check it out.