How Fast is Your Turnaround Time? 418
petrus.burdigala writes "I work for a mid-sized commercial software company (~20 Mloc) and we are frequently challenged by our supervisors to get fixes around the clock. Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment), which I consider not so bad. But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic. So I wanted to get feedback from my peers: are we doing that bad? It takes months for other software vendors to issue zero-day exploit fixes, are our customers being unreasonable?"
1 to 2 weeks (Score:5, Informative)
My jaded perspective... (Score:5, Informative)
Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment)
Not unreasonable, depending on the size of your release. (How many modules and how many LOC you're changing, the number of change requests or bug reports in the build).
But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.
I think they're smoking crack.
So I wanted to get feedback from my peers: are we doing that bad?
With your regular release schedule, I don't think so.
are our customers being unreasonable?
Yes. That's what they do. If they want a crash development program to get this "patch" out the door that fast, they seriously risk software which does nothing but crash. Really, if they want it that bad, they run the risk of getting it that bad.
You have to ask yourself and your "support team" (sounds more like marketing to me): "Do we wish to ruin a perfectly good reputation for quality and reliability in one hurry-up bashfest followed by weeks of agonizing on-line debugging?" Really, advocate any kind of work-around and risk mitigation response before being pushed into an overly-hasty release that will linger on your reputation like a dead skunk.
Two prong approach (Score:4, Informative)
(1) Get the customer up and running again as fast as possible. This is as often as not some sort of workaround that is not pretty, nor is it permanent, but it works. The workaround does get thorough testing (impossible within the time frame) but the customer is aware of this and willing to accept the risks.
(2) Get the customer a proper, version controlled, patch that they can install to fix the problem permanently. This can take weeks, most of that time being testing. If the customer is insistent we will ship them the proper patch before it is fully tested (again, making them aware of the risks) and continue testing so that we can send the customer some warm and fuzzy news later on (or, if we find a problem, another patch).
Make a policy. Stick by it. Make it reasonable. (Score:3, Informative)
Criticals get next attention when show stoppers are out. 48 hours, depending on interdependencies and QA needed to make it work; it's not part of an official stable code tree until later.
Minors are in the next stable branch release; every whatever you can handle.
Nigglies are changed when the stable branch releases.
Don't deviate from your policy, and make sure the sales people KNOW AND UNDERSTAND what this indicates and implies. No exceptions; see above.
Re:It depends... (Score:2, Informative)
Most of my cases are resolved by explaining to the customer things that are unclear in the documentation, so it's unusual to decide within 48 hours that the customer is reporting a real bug. Once we agree that they are, then I can usually reproduce the behavior in a day. Once reproduced, then we do not consider 2 to 5 days for a fix to be delivered to be out of line.
Questions I can answer same day. Fixing bugs takes time.
Re:The Real Meaning of Bad (Score:5, Informative)
If they ask for something within 48 hours and expect something usable, it is up to you to educate them.
Re:Parent is right. (Score:1, Informative)
How we handled this in one environment was simple. All normal updates followed process to the tee or else.
But a process deviation authorization process also existed. In essence, the VP of engineering and marketing, the customer and legal had to sit down and sign a binding contract to get pre-release. The contract basically said "Without duress or obligation we both agree that we are not responsible for ANY issues or fitness for use and are not obligated in any way to mitigate. You cannot disclose any issues you may have with a pre-release product as it is of alpha-beta test quality and acknowledge this." Participated in a few too. Usually only done when the customer had some competative advantages by accepting the added risk.
Essentially an agreement that if say, not being tested enough it wiped out a 10TB DB tough O. If not signed, they had to wait until Q/A said cut, wrap and ship.
Re:Parent is right. (Score:5, Informative)
If a customer buys a support contract that explicitly states that 1 week is a reasonable turnaround time for an issue you'll be amazed to find out how pleased the customer is when you fix a problem in 72 hours. If some asshole salesman tells them that they can expect solutions to any issue in 2 hours, well, get ready to deal with an "unreasonable customer".
I unreasonably expect this post to be modded +5 insightful.
Re:That works both ways. (Score:2, Informative)
Re:Turnaround time (Score:3, Informative)
We generally get fixes for real bugs out within 24 hours
We do too, although we release every two weeks, so it would have to be a critical security bug to get us to do a patch release and not just include the fix in the next code drop. Genuine defects (as opposed to platform support issues, production support issues, configuration problems, or changed requirements) are a stop-the-line issue.
Um, and if it's taking you a month to fix a problem, what is your level of automated test coverage? You are doing TDD, right? And how much time are you spending on refactoring for maintainability?