Slashdot is powered by your submissions, so send in your scoop


Forgot your password?

How Fast is Your Turnaround Time? 418

petrus.burdigala writes "I work for a mid-sized commercial software company (~20 Mloc) and we are frequently challenged by our supervisors to get fixes around the clock. Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment), which I consider not so bad. But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic. So I wanted to get feedback from my peers: are we doing that bad? It takes months for other software vendors to issue zero-day exploit fixes, are our customers being unreasonable?"
This discussion has been archived. No new comments can be posted.

How Fast is Your Turnaround Time?

Comments Filter:
  • Definition (Score:5, Funny)

    by ShawnCplus ( 1083617 ) <> on Tuesday November 13, 2007 @05:18PM (#21341633) Homepage

    aren't our customers being unreasonable

    It may just be me but I think that's why they are called "customers"
    • Parent is right. (Score:5, Insightful)

      by iknownuttin ( 1099999 ) on Tuesday November 13, 2007 @05:26PM (#21341781)
      It may just be me but I think that's why they are called "customers"

      Sometimes, customers are unreasonable and if they are, they should be treated with respect and the problem explained to them. Yes, they may be incredulous, but if you hold your ground (if they're being unreasonable), treat them with respect, they will come around.

      The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.

      • by khasim ( 1285 ) <> on Tuesday November 13, 2007 @05:42PM (#21342043)
        Maybe the customer is being unreasonable.

        Maybe the developer is being unreasonable.

        It isn't possible to determine which from either person's viewpoint. You will ALWAYS think that you're right and that the other person is unreasonable.

        Which is why you need criteria for bug escalation. Generating an incorrect response on 1 type of transaction for 1 specific scenario that may pop up once a year is far less important than a bug that corrupts the entire database.

        And if your product is considered "mission critical", I would expect a data corruption bug to be fixed within 24 hours. Even if it is nothing more than rolling back the recent patches and re-issuing the previous version.
        • by log0n ( 18224 )
          Yeah.. I was going to say.. I interpreted the FP as saying they are the customer, so they have every right to be 'unreasonable'. They bought something (seems to be mission critical) so they should have every right to expect it works for them. That's how the customer/provider relationship works, especially one that provides dedicated and one-off services like this seems to indicate.

          Of course, 48 hour turn around may not be possible.. that's another issue.
        • by Laglorden ( 87845 ) on Tuesday November 13, 2007 @05:55PM (#21342237) Journal
          The problem is when the customer is being unreasonable, the "support" (or more likely sales) just agrees to everything they say and "sure, we'll fix it" because they don't a) know any better b) they wan't to sell, not take the conflict c) they're stupid and just passes this backward "fix this, NOW!"

          Then you're going to have a bigger problem! It's the same thing in any kind of relationship, just bowing and scraping and always saying "it's my fault" is going to cause bigger problems in the future than just saying "nope, we're not gonna fix that. or "sure, well fix it, but not now, you'll get your patch when it's tested properly, in the meantime, do this instead"
          • by Anonymous Coward on Tuesday November 13, 2007 @06:13PM (#21342485)
            The battle between sales and support has raged for years. Forget sunni v. shite, red v. blue, Kobe v. Shaq. Sales v. support is the ultimate battle. Sales tells the customer what they want to hear. Support tells the customer the truth. I've worked both sides of the aisle. As a sales rep I try to be honest about what I'm selling. The software I sell does not support P2P. We don't support it. Need a server. I let the customer know. Some of the other reps kind of gloss over it really fast just to make the sale. Then the customer calls back to support upset. I'd rather avoid that by being honest up front.

            Now in this case the customer may be being unreasonable with the 48 hrs demand. But it all depends on the issue. There have been times when my company has been able to get out a quick fix within 24 hrs and other times when it has taken 3 weeks. It all depends on what the issue is, and what the solution is. There has to be some kind of middle ground that could have been reached.
        • by arth1 ( 260657 ) on Tuesday November 13, 2007 @06:11PM (#21342463) Homepage Journal

          Which is why you need criteria for bug escalation. Generating an incorrect response on 1 type of transaction for 1 specific scenario that may pop up once a year is far less important than a bug that corrupts the entire database.

          Bullshit. A corrupted database of inventory of toilet paper can be far less dangerous than the one type of transaction for one specific scenario that may pop up once a year when that one transaction decides the rod position in a nuclear reactor.
          In other words, you have to know the importance to the customer.

          And, yes, weeks can be way too slow, depending on the nature of the bug. If it means a large customer's ordering system is down until the patch is ready, not fixing it for weeks is likely to lose you the customer and any goodwill with a lot of other companies.

          No, you don't need criteria -- they will get in the way of common sense every time.
          What you need is rapid impact analysis, and teams that are able to tackle different tempos, based on what others better informed tell them.
          • by Fulcrum of Evil ( 560260 ) on Tuesday November 13, 2007 @06:34PM (#21342727)
            You sell toilet paper and control reactors with the same machine?
            • by Jesus_666 ( 702802 ) on Tuesday November 13, 2007 @08:41PM (#21344117)
              You have no idea how the real world works.

              Sometimes I long for the easy days before I got that PhD in Nuclear Science/Tissue Engineering. Before I knew how the nuclear power/toilet paper industry really works.
              • Re: (Score:3, Funny)

                by Sanat ( 702 )
                Mistakenly install the roll backwards just one time and you will never work again in the toilet paper/nuclear power industry. They don't forget nor forgive. They just wipe up the mess and insert the rods again.
          • by gstoddart ( 321705 ) on Wednesday November 14, 2007 @12:13AM (#21345845) Homepage

            Bullshit. A corrupted database of inventory of toilet paper can be far less dangerous than the one type of transaction for one specific scenario that may pop up once a year when that one transaction decides the rod position in a nuclear reactor.
            In other words, you have to know the importance to the customer.

            You know what? To the customer, they're all nuclear rods. They don't care about your problems, they care about theirs. And those problems are critical to them.

            You wanna know how escalations work, find out how important the customer is to the company. If you can get someone to unreasonably escalate your call because you're a big contract, you can get a lot of attention. If you're a small customer, or, if the vendor has balls, you might have to wait. Because what you're asking for isn't that big of a deal to us.

            In my experience, if you can convince a VP this is a show stopper, you can get a lot of screaming -- it's like being in the army that way. If you can't get a VP/VIP on board, you get to stand in line.

            Your priority to us is in proportion to related revenues to you. :-P That's how corporations work.

            Unfortunately, a fix can't always be done in the timeframe demanded. Sometimes, you have to push back. Sometimes, you don't get a choice. :-P

          • Re: (Score:3, Insightful)

            by mwvdlee ( 775178 )
            I'm guessing that toilet paper production company values it's database quite a lot.

            You're not talking about "importance to the customer" but rather "importance to society".

            Although a nuclear reactor is kinda important, there is one minor problem with reasoning this way; pretty much everything is more important than toilet paper inventory from society's point of view, so their inventory bug will never be fixed.
        • Re: (Score:3, Interesting)

          by meatspray ( 59961 )
          The OP really doesn't give enough detail to give decent advice, the bug could be huge or tiny. The system could be unnecessarily complex or just gigantic.

          4-5 weeks for a maintenance patch is fine, but if this bug is stopping your client from doing their work, you should be getting single fixes out the door much faster than that. Patches need to be prioritized. Go back to your last rev, fix only the problem in question and release that small subset as a hotfix.

          IMO unless you're writing an operating system,
      • Re:Parent is right. (Score:5, Informative)

        by jomama717 ( 779243 ) * <> on Tuesday November 13, 2007 @06:17PM (#21342527) Journal
        The most valuable skill I learned during my short time as a field consultant was how to "manage expectations" (pardon the bullshit bingo term). It's not the customer that is being unreasonable, it is that they have somehow adopted unreasonable expectations of what you can provide them. In other words, it's all your company's fault.

        If a customer buys a support contract that explicitly states that 1 week is a reasonable turnaround time for an issue you'll be amazed to find out how pleased the customer is when you fix a problem in 72 hours. If some asshole salesman tells them that they can expect solutions to any issue in 2 hours, well, get ready to deal with an "unreasonable customer".

        I unreasonably expect this post to be modded +5 insightful.
        • by TheThiefMaster ( 992038 ) on Tuesday November 13, 2007 @07:36PM (#21343419)

          I unreasonably expect this post to be modded +5 insightful.
          Which is of course why you got +5 Informative.
        • Re: (Score:3, Insightful)

          by ninjagin ( 631183 )
          You make a great point.

          I'd add that if you are really good at turning around fixes in 72 hours, the customer will come to expect that. It will get to the point that they'll growl and pester when you take 96 hours on a fix.

          Managing the expectations generated by your history of success is much harder to do, regardless of what the SLA says.

        • by einhverfr ( 238914 ) <> on Tuesday November 13, 2007 @09:04PM (#21344291) Homepage Journal
          Now, there *are* unreasonable customers out there, but I agree with you. Most of the time it is an issue of expectations.
        • Re: (Score:3, Interesting)

          by chathan ( 714599 )
          We used to have 2 weeks turn around time for critical fixes. So it is always better to inform the customer that 2 weeks is the time they can expect the fix even if the bug is way above critical status. But there were issues which make few features unusable for the customers and we know that they are heavily dependent on these features.We try to fix it in a week or 3 days time.

          The important thing is your customer facing person(for that matter your manager) should be aware that even if he or she thinks the b

      • by einhverfr ( 238914 ) <> on Tuesday November 13, 2007 @06:23PM (#21342597) Homepage Journal
        For security issues we usually have a 48-hr to 1-month turn around time. I think a turnaround time for urgent issues of more than a month is excessive. Minor issues get fixed based on a number of factors and the turnaround time varies from 24 hours to 3 months.

        In case you are wondering why the floor on the security issue fixes is a little longer, we usually put the initial problem and the fix through extra review so we ensure we truly understand the problem and that the fix solves all likely related issues. Then we have to decide whether to do a patch or a maintenance release. This process adds additional time.

        Having said this, it is impossible to know from the description whether the customer is being reasonable or not. I have had issues where I had to come up with a fix within 24 hours and other cases where the demand was unreasonable. Hope this helps.

        1) What is the business impact of the bug?
        2) What is the data integrity impact of the bug?
      • Re:Parent is right. (Score:5, Interesting)

        by vertinox ( 846076 ) on Tuesday November 13, 2007 @06:26PM (#21342647)

        Yes, they may be incredulous, but if you hold your ground (if they're being unreasonable), treat them with respect, they will come around.

        The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.
        I wouldn't say its not contempt but you get what you pay for. I once worked for a small software company who would push out emergency patches (even if the issue was minor) within 24 hours if you had the $10,000 package including paid support. No questions asked. Heck, we'll fly one of our reps out there to install the patch himself.

        If you had a single license and no paid support... Well... We might have a general update next month with a public patch. We might not. Have a nice day.

        Of course when you sell software as a service then thats how it works.

        As a side note, one customers feature request created a completely separate build just for that customer which was annoying to the programmers but since they paid good money for it, they got what they asked for. Although... I remember the programmers eventually including the features for everyone else as a optional package just to avoid that so in the end even the single client customers benefited.
      • by pla ( 258480 ) on Tuesday November 13, 2007 @09:03PM (#21344287) Journal
        Sometimes, customers are unreasonable

        "Sometimes"? Heh... Good one.

        and if they are,

        "if"? Man, where do you come up with this stuff?

        they should be treated with respect

        Ahahahaahahhaaaa... Heh...

        and the problem explained to them.


        Ahem... Please, stop, I can't take anymore.

        The fact that the parent was moderated down just shows me that the arrogance, contempt, and stupidity in corporate America is alive and well - especially in IT.

        Some people deserve contempt and our scorn.

        They act as though we can save the world before dinner when they want something, and call us miserable worthless slacker bastards the next. They insist we fix their problem in 48 hours when they can't even describe the problem accurately enough to reproduce. They need us and beg us for help and resent every second of it. They treat us like disposeable/interchangeable cogs, then bemoan that we each have unique and difficult-to-replace skillsets.

        You want to know why geeks look at most people with utter contempt? Because they spit on us first.
        • Re: (Score:3, Insightful)

          by NekoXP ( 67564 )

          Some people deserve contempt and our scorn.

          They act as though we can save the world before dinner when they want something, and call us miserable worthless slacker bastards the next. They insist we fix their problem in 48 hours when they can't even describe the problem accurately enough to reproduce. They need us and beg us for help and resent every second of it. They treat us like disposeable/interchangeable cogs, then bemoan that we each have unique and difficult-to-replace skillsets.

          You want to know why

          • by pla ( 258480 ) on Wednesday November 14, 2007 @06:34AM (#21347685) Journal
            You need to respect your customers, because if they went away, you'd be out of a job, it's that simple.

            Technically true, but irrelevant. If cows went away, we couldn't have any more hamburgers. That doesn't mean we'd all starve to death, because we can eat other things. But you want know the funny part here?

            We could do most of their jobs (perhaps with a bit of training). Not all of us, and not all of their jobs, but in general. They cannot, ever, learn our jobs. One of our surprisingly few actual skills, "problem domain reduction" (kudos to 19thNervousBreakdown for the term), most people simply can't learn, regardless of will or even intelligence. On the flip side of that, however, it means we can pretty much accomplish anything we try, from coding to plumbing to animal husbandry to stonemasonry.

            Think, really think, about how many geeks you know who, during the tech crash half a decade ago, did just fine on a variety of completely unrelated-to-IT jobs. Personally, I did a stint in construction/carpentry, and produced some damned nice work (if I do say so myself) - with ZERO training beyond casual observation of standard proceedures. I don't say that to brag - Hell, I don't really consider it much to brag about - Just putting myself forward as an example. We can do their jobs. They can't do ours.

            No, it's because you act like a self-important little shite who thinks they should be bowing on their knees and sucking your dick for every line of code you produced.

            Not really, because I code for me. They just pay me for it. I'd do it in my spare time if I didn't do it as part of my employment. I stay up-to-date on the world of IT because I find it fascinating, not because someone pays me to freshen my skill-set or because the terms of my state-permission-to-practice requires some pathetically low number of hours of study per year.

            Anyway, all of the above said, I do try my best to remain humble and polite to most people, geek or not - And for the most part, I succeed. I very much doubt most people who know me would call me a "self-important little shite". But still, the constant jabs come anyway - From "complimenting" me on my skills the same way you would compliment medusa on her hairstyle, to barely-tempered insults only blunted by the fact that we've usurped the language no differently than blacks using the word "nigga". "Dude, you're such a geek!" "Yeah, thanks". People look at us as freaks for what we can do, and you tell us to respect them?
  • by Hanging By A Thread ( 906564 ) on Tuesday November 13, 2007 @05:18PM (#21341635)
    How much of that 48 hour deadline did you waste reading /.

    Get back to work!
  • by Black-Man ( 198831 ) on Tuesday November 13, 2007 @05:19PM (#21341653)
    You have to serve the client who is paying the bills - and we had a very vocal one (Nik*). We had a running joke about the release d'jour. But it wasn't a joke. We literally would push a new build to them every day which contained minor bug fixes. It was maddening! But no one had the balls to stand up to the 800lb gorilla, so the madness continued. As a side-note, they were acting as a beta tester and anyone in the software business knows what that can mean.

    • by soloport ( 312487 ) on Tuesday November 13, 2007 @05:39PM (#21341995) Homepage
      Our running joke used to be:
      Marketing: We need it real bad!
      Engineering: How bad do you need it?
      Marketing: <puzzled look>
      Engineering: Careful what you wish for... OK, Ops. Ship it!
      • by clsours ( 1089711 ) on Tuesday November 13, 2007 @05:45PM (#21342093)
        If they ask for something within 48 hours and know what that means, then they deserve what they get.
        If they ask for something within 48 hours and expect something usable, it is up to you to educate them.
        • It's about risk (Score:5, Insightful)

          by PIPBoy3000 ( 619296 ) on Tuesday November 13, 2007 @06:53PM (#21342927)
          I work for a large healthcare organization and typically have very fast turn-around times (bugs often get squished within an hour). For clinical applications and other core applications, though, we're much more methodical and careful.

          I often explain to the user that I can push changes out immediately, but it introduces certain risks. I then detail the risks they may face, and that if they say to go ahead anyway, at least they'll be aware of what might happen.
    • by Anonymous Coward on Tuesday November 13, 2007 @05:50PM (#21342163)
      I work in a small company doing support (amongst other things). I've been there longer than most of the developers due to high staff turn over.

      A common scenerio for me goes like this:

      1 - client reports a problem.
      2 - spend 2-3 hours on phone with client identifying what's really going on.
      3 - spend an hour or so in SQL Profiler/Delphi debugger to find the root cause.
      4 - half an hour documenting what the problem is, causes & how to replicate in order to hand it off to a developer in an off-shore office

      ... wait a week or so for the client to get really cranky ...

      4 - have my supervisor ask me Monday morning if I can look at it because the client is cranky & the developer is sick/has quit/has family crisis/there is a public holiday in that country.
      5 - fix the damn thing which only takes a few hours to code & test & delivery after all the hard work was done in step 3
      6 - wonder why the hell I am still with this company
      • Re: (Score:3, Insightful)

        by jdjbuffalo ( 318589 )
        You can get a better job out of that most of the time.

        Sit down and detail what you REALLY DO for the company and how valuable you are. Explain that you would like to get a new job title and pay raise commiserate with your responsibilities. If they don't pay or you think they may fire you then start looking for a new job.

        Life's too short to not get paid what you're worth. Plus you can retire earlier. :)
      • by PitaBred ( 632671 ) <> on Tuesday November 13, 2007 @08:05PM (#21343741) Homepage
        Consider that week or so of waiting as a chance to learn a bit about business politics, your company's political hierarchy and start getting things changed. If they don't change, leave. You're the only reason they're staying afloat, if everything technical eventually falls back to you.
  • 3 hours right here. Maybe you just need to rub your hands in some comcast coax for extra speed (k thx lame comcast commercial ftw!)
  • Hmmm. (Score:5, Funny)

    by Stanistani ( 808333 ) on Tuesday November 13, 2007 @05:20PM (#21341665) Homepage Journal
    What was that exploit again?
  • 1 to 2 weeks (Score:5, Informative)

    by duplicate-nickname ( 87112 ) on Tuesday November 13, 2007 @05:20PM (#21341669) Homepage
    For high priority bug fixes, it usually takes 1 to 2 weeks to get a patch out once we determine that a patch is needed.
  • by LiquidCoooled ( 634315 ) on Tuesday November 13, 2007 @05:20PM (#21341681) Homepage Journal
    It depends upon the nature of the problem and the competency of the developers.
    If you know enough of the code tree you can tell when first reproducing and examining the failure whether it is a one off mistake or a larger procedural fault.

    Single instance stupid errors (doh! moments) can be rectified and put through testing fairly quickly, however if your initial examination uncovered a larger problem then obviously the process will take longer (if at all - consider workarounds).

    If the original dev/test team has been replaced over time this becomes a more difficult issue and every bug must go through complete verification simply because the extent or ramifications of the code modification will not be known.

    In some instances we have had fixes out of the door the same day an issue was noticed, in others months go by before a final fix is put in place.
    • I love the parent's subject-line analogy.

      I'd add, it depends on product, the complexity of the codebase, the extensibility, modularity, readability, and extensibility of the codebase (eg, if it's highly modular it's easier to test a fix that's limited to the module/plugin)

      I'd suggest that weeks sounds too long for an in the wild update without a security patch - or published workaround limiting your exposure. (eg, "use this method to restrict the IPs that can access it to trusted ones.") But that isn't m
  • four or five WEEKS? (Score:4, Interesting)

    by tannhaus ( 152710 ) on Tuesday November 13, 2007 @05:21PM (#21341685) Homepage Journal
    I can understand a week, but honestly...if you're leaving your customers vulnerable for over a month, they might start looking elsewhere

    Exploits should be a high concern for any company
  • of the issue. My team has had to get things out the door in 4-6 hours before. Just make sure you have people that have intricate knowledge of the app so that you can properly scope any changes, which allows you to perform good QA.

  • by netsavior ( 627338 ) on Tuesday November 13, 2007 @05:21PM (#21341705)
    I work for a bank so we don't do box software, but our patches have to meet FTC standards and Federal bank standards.

    It is uncommon, but not unheard of to have an 8 hour fix. In cases of customer data vulnerability, legislation has been made such that if we are aware of a problem, we have an automatic injunction against us continuing to do business unless the problem is resolved. So when we have a security flaw, our bank stops working untill it is fixed. So yeah 48 hours would have people fired for sure.

    Compliance/security are the only two things that can spark a release with less than 72 hours notice though.
  • by Wellington Grey ( 942717 ) on Tuesday November 13, 2007 @05:23PM (#21341727) Homepage Journal
    But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.

    Well, we really can't answer that question with knowing how big the problem is. If it's an embarrassing typo on a dialog box, then 48 hours is reasonable. If it's a windows vista security patch, then 48 days would be unrealistic.

    -Grey []
    • If it's a windows vista security patch, then 48 days would be unrealistic.

      Not to sound like a M$ fanboy or anything, but they're average time to fix is 48 hours. The other 46 days is just how long it takes them to cycle it out. I assume that if you're cooperate or some other important customer it is possible to get the patch faster, but I honestly have no idea.
    • by techno-vampire ( 666512 ) on Tuesday November 13, 2007 @06:18PM (#21342541) Homepage
      If it's a windows vista security patch, then 48 days would be unrealistic.

      It doesn't take 48 days to burn a Linux CD and send it to the client.

  • Can't compare (Score:5, Insightful)

    by Sloppy ( 14984 ) on Tuesday November 13, 2007 @05:24PM (#21341735) Homepage Journal

    It depends on what you're maintaining and how complicated it is. I've gotten fixes out in 2 or 3 minutes. That doesn't mean I'm fast and you're slow, though. "How fast is your turnaround?" is like "how long does it take to write a computer program?" It's hopelessly vague.

    • Re:Can't compare (Score:5, Insightful)

      by Shagg ( 99693 ) on Tuesday November 13, 2007 @05:50PM (#21342151)
      Exactly... "How long is a piece of string"?

      In addition to all of the other comments about the scope of the problem, number of resources, etc, you also have to take into consideration what you're changing. Obviously there are huge differences between patching the avionics system on an airplane vs a banner ad on a website. I've given estimates anywhere from hours to months before. There's no such thing as "X is the right amount of time for a patch" without a lot more details.

      One thing you can always do is try and work with the customer to make them aware of the issues. You can tell them that it's possible to get a patch out to them faster, but you will be skipping a lot of the QA in order to do so (depending on what flexibility you have with the standard company process). The risk of it failing would be theirs. If they're OK with that, then you might be able to expedite things. It all depends on what you're patching and how important it is to them.

      • Re: (Score:3, Funny)

        by Applekid ( 993327 )

        "How long is a piece of string"?
        Why, twice the distance from its midpoint to an end. Pfft. Easy. ;)
  • Depends on alot of things: the size of the shop, the size of the project, the size of the fix, etc. Do you have an MVC framework? Do you have a centralized database architecture?

    The better engineered the project is, the faster development will be. The worse engineered and less centralized, the harder it will be. Also, the larger the size of the project, the more time it will take. Also the size of the company adds extra time as well; smaller shops can fudge on steps to get a fix out but larger shops have

  • Management strategy (Score:5, Interesting)

    by gurps_npc ( 621217 ) on Tuesday November 13, 2007 @05:25PM (#21341771) Homepage
    There is a management strategy that basically says aim for the stars.

    Yeah, your turn around time seems good and yes, the customer's request is beyond industry norm.

    That might mean one of three things:

    One: Customer is being foolishly optimistic.

    Two: The entire industry is bad about turn around time, and can, if pushed improve it to 48 hours.

    Three: Customer needs it really quick and is hoping to get it quicker by asking. They know 48 hours is well beyond the norm, but are hoping you can do it anyway, because the more time it is unpatched the more they are screwed. They know that if you don't ask, you can't get, so they are at least 'asking'.

    Me, I think it is a combination of all three. Customer is being a bit optimistic, the industry is bad about turn around time, and also the customer knows it is a bit optimistic but is making the request anyway in hope you will provide amazingly good service.

  • Unrealistic (Score:5, Insightful)

    by gweihir ( 88907 ) on Tuesday November 13, 2007 @05:27PM (#21341785)
    With a little simplification, you have four parameters: Difficulty, quality, speed and available resources. Whenever you fix three, the fourth follows (with some unvertainity). It is well known, that there is a limit on how much you can improve the speed with more resources. So there is an upper limit on speed already. The second problem that difficulty is unknown when starting such a task. There is no fix for that.

    So if these people fix speed and available resources, and difficulty is fixed by the task, quality is determined by these factors. Period. There is no arguing with hard, real limits. If they do also want to specify the result quality, then they have to leave speed open. Again, there is no way around that limitation. In fact they should be happy if the team manages the required quality at all in reasonable time. Not all teams do.

    Maybe thisn will be an argumentation that is inderstandable for people with a business background. Engineers should already know this.

    Software engineering is engineering. Engineering tasks in general have minimal time requirements. Look at structural engineering: Nobody would try to design and build a full-custom bridge in a week. Instead it takes up to a decade, depending on difficulty. And you can generally not speed things up by increasing the team size.

  • by idontgno ( 624372 ) on Tuesday November 13, 2007 @05:27PM (#21341789) Journal

    Overall, we manage to get a 'bullet-proof' patch in about 4-5 weeks (from coding->QA->Build/Packaging->shipment)

    Not unreasonable, depending on the size of your release. (How many modules and how many LOC you're changing, the number of change requests or bug reports in the build).

    But the other day, we got an urgent request from our support team to come up with a decent fix in 48 hours. I think they're a tiny bit unrealistic.

    I think they're smoking crack.

    So I wanted to get feedback from my peers: are we doing that bad?

    With your regular release schedule, I don't think so.

    are our customers being unreasonable?

    Yes. That's what they do. If they want a crash development program to get this "patch" out the door that fast, they seriously risk software which does nothing but crash. Really, if they want it that bad, they run the risk of getting it that bad.

    You have to ask yourself and your "support team" (sounds more like marketing to me): "Do we wish to ruin a perfectly good reputation for quality and reliability in one hurry-up bashfest followed by weeks of agonizing on-line debugging?" Really, advocate any kind of work-around and risk mitigation response before being pushed into an overly-hasty release that will linger on your reputation like a dead skunk.

  • Seriously... but only once.

    We were on-site at the customer, and we were involved in a shootout to see who would get a major contract.

    We were "pre-demoing" to a bigwig, and it turned out we had an incorrect concept of one of the requirements. Hacked out a fix in 20 minutes.

    Needless to say, the customer was impressed (as were my bosses :P).

    For production level code, though, I'd never do that, but it was a do-or-die sort of thing.

    Oh, and we won the shootout (it wasn't even close, according to some of the guys
  • by techpawn ( 969834 ) on Tuesday November 13, 2007 @05:27PM (#21341801) Journal
    A patch (IMHO) is a bug fit to existing code. Given the resources we should be able to get a PATCH out in a week. However, if you need a new version of the software to address the issue. Then we're talking longer development/testing/QA times if which case 4-5 weeks would not be unreasonable. Bugs should be fixed as soon as they are spotted. If their is need for a whole rewrite then you may want to talk to your staff
  • Pick 2 of 3 (Score:3, Insightful)

    by CambodiaSam ( 1153015 ) on Tuesday November 13, 2007 @05:28PM (#21341819)
    I know I'm going to end up baiting some developers, but I work for a specialized ASP and see a ton of third party software from a perspective few get...

    Normally, the smaller the company the more agile. No surprise. They also get patches out faster too. Also no surprise.

    When we look at vendors of equal size, the ones who are really quick at sending out patches are in that situation because their software is more buggy, and they have a *lot* of practice. It never fails.

    In response to your question, I would suggest that you should look more at the frequency of patches and less at the duration. Sure, it might not be as fast as your support group wants, but if you start reflexivly sending out patches every time someone yells, then your overall product will suffer since you can't possibly do the proper QA to ensure THAT patch you just whipped up doesn't break something else.

    That brings me to the age old choice:

    Pick 2 of the following:
  • by Joe The Dragon ( 967727 ) on Tuesday November 13, 2007 @05:29PM (#21341825)
    How much time do you spend on TPS reports?

    The last time I did one I forgot the cover page and my 7 bosses all bugged me about it.
  • What was the issue?
    Some things are trivial and fairly obvious mistakes, that are often easy to fix without breaking anything, in which case it's not too hard to get a patch out quickly. Format string problems for instance, shouldn't be too hard to correct.
    Often a little extra validation can correct a problem too, just put in a check for a value being zero before doing a division etc. Things like this are also easy to test, and don't break anything else.

    Longer turnarounds arise when the issue is more of a de
  • Extreme Programming (Score:2, Interesting)

    by obduk ( 1154583 )
    Ok, the name might suck, but the company I work at follows the Extreme Programming practice, a kind of agile programming. I have only worked there a few months, and had never herd of XP before, but am now converted. We work in pairs, which instantly adds a whole testing level. Deployments of code are done once every week, but sometimes more in an emergency. We write code test first, then run a build on our machines, then we upload it to a test environment where automatic tests are run. Finally on passing th
  • In the case of a zero-day exploit, 48 hours is an entirely reasonable expectation. As far as I can tell, it is far from the norm, however. For less egregious flaws, feature upgrades, or normal bugs, a slower turnaround would be perfectly reasonable (and what I'd expect). But for a zero day exploit... Well, I firmly believe software companies need to be held more accountable for those by their customers. If that's what your customers are doing, then good for them.
  • by seebs ( 15766 ) on Tuesday November 13, 2007 @05:31PM (#21341871) Homepage
    At BSDi, the initial patch (which did have flaws, but it fixed the problem) for the f00f bug was same-day, I believe; might have been next-day, depending on where you're counting from. (Contrary to popular belief, this didn't violate any NDAs.) Now, that was an emergency patch -- it took a while to come up with a patch that fixed the bug without noticable ill side-effects.

    We had a better patch later, but the initial emergency patch was VERY fast.

    On the other hand, if the initial bug report is "Sometimes the program hangs, no, I don't know when. Maybe every week or two." -- well, that's gonna be hard. Exploits generally have the advantage that an exploit is by nature at least somewhat reproducible, and the hardest part is often getting a reproducer. I've had it take six hours to develop a usable reproducer, and three minutes to develop a patch.

    Release time depends hugely on process and procedure. IMHO, an ideal procedure would have some kind of way to get a Temporary Patch out into the field ASAP when there's an exploit.
  • D'oh, it would be helpful to know which kind of patch we are talking about. Is that security? If so, how critical? Is that a theoretical case, or a gaping hole that is a dead giveaway to any script kiddie? Is the problem just annoying or mission-critical? Is that going to be used in a network? As a server service? On which platform? If it is a new feature, does it require re-engineering of the code base, or is it a drop-in feature you can be over with in half an hour?

    Most importantly, is your code base wel

  • It depends... (Score:5, Insightful)

    by gillbates ( 106458 ) on Tuesday November 13, 2007 @05:31PM (#21341883) Homepage Journal

    48 hours is tad bit tight. However, I've turned things around in a similar amount of time.

    But, the old adage is true: you get what you pay for:

    • Granted, 48 hours is tight, but possible if you know the root cause, *and* the customer is willing to forego the usual QA process before delivery. It doesn't mean that you don't do QA, but rather, that you do it later and patch the patch, if necessary. In most corporations, this means that if the customer doesn't complain, QA doesn't get done for these "extra-special" releases.
    • Four to five weeks for 20Mloc is probably about average. As a general rule, a good team will average about 1 week per department the fix has to go through: field team -> engineering (fix) -> department review -> QA review -> ship. However, in some organizations, particularly the smaller ones, defects can get turned around in 1 to 2 weeks, especially if the customer works directly with the engineer/developer, and the developer is authorized to make releases to the customer. Be aware that your customers may have dealt with such organizations in the past.
    • 20Mloc is not really that big, provided that the project and code was well-organized from the beginning and the original developers are still on staff. But, if this is not the case, or you have a large developer base, where no one is actually an expert on the systems, or subsystems, you can add about a 50% overhead to your turnaround time. If the original developers are no longer there, add 100%.
    • If you don't know the root cause of the problem, you can't promise anything, and you need to inform the customer of this - you simply can't make any guarantees because you don't know the scope of the problem. Once the root cause has been identified, a 48 hour deadline is still tight, but is long enough to allow the key developer to build and do some rudimentary testing of the fix. Should the customer choose to accept it at this point (and you *must* make the point that it is their choice), they must be willing to forego the normal QA process, and should sign a statement of understanding to that effect.

    When faced with unreasonable deadlines in the past, I usually voice my opinion once, and just do the best I can. Your higher-ups are probably already quite stressed at this point, and adding stress to the situation doesn't do anything for your career or theirs. Rather, if you make the point that you're doing the impossible, you might just have a little bit more bargaining power when it comes time for raises.

    But on the flip side of the coin, if management doesn't learn, and you find yourself constantly asked to do the impossible, you might want to consider employment elsewhere...

    • Re: (Score:2, Informative)

      by Weasel Boy ( 13855 )
      I work as a tech support engineer, and if I could mod the parent to +5 (Insightful), I would.

      Most of my cases are resolved by explaining to the customer things that are unclear in the documentation, so it's unusual to decide within 48 hours that the customer is reporting a real bug. Once we agree that they are, then I can usually reproduce the behavior in a day. Once reproduced, then we do not consider 2 to 5 days for a fix to be delivered to be out of line.

      Questions I can answer same day. Fixing bugs ta
  • In general, the company I work for goes from end of development to GM in about a month, for boxed CD software that ain't bad. That assumes QA has been testing right through development. But it's going to vary widely based on the industry and size of the product.

    I blogged about the opposite issue this morning. []

    Essentially it comes down to choosing a smart policy and sticking with it. If your company's policy is "Spend 2 weeks QA on every relea
  • If you are talking about serious security vulnerabilities with-out a reasonable work around, then the customer has a reasonable expectation that you literally work around the clock to fix the issue.

    If its a "normal" bug then I think several weeks or months is more the industry standard.
  • Let alone the importance of the problem.

    That is why I am having trouble giving you a fixed number. I have been involved in fixes done in days and ones that take months. Context.

    The real questions is

    Does your management know how to prioritize tasks?

    Then, do they know how who to put to to each task?

    Finally, do they know how to accurately judge if the work is done correctly?

    The problem I have always encountered is that a needed fix gets priority at first, then pet projects somehow get in there, and the deadl
  • by mcmonkey ( 96054 ) on Tuesday November 13, 2007 @05:35PM (#21341935) Homepage
    You really have to supply some more detail to get any useful answer. And what is ~20 Mloc? About 20 million locations?

    What kind of software? What classifies an urgent request? Do you make games, and an urgent request means your bug just made front page /.? Do you make internet-facing apps, and an urgent request means your customers just formed a spamming bot-net? Are in the medical/health care field, and an urgent request means folks might die?

    I think a better question is, how do you classify bugs? How do you make that trade-off between fixing a bug ASAP and taking the time to make sure the bug fix is done right?

    Who is involved in the decision process? Is it just the technical & regulatory folks? Do you pull in business folks to help gage customer impact? Do you pull in sales and support to see if they can push a work around before the final fix is ready?

    Those are all better questions than, "How fast do you do this task of unspecified scope."
  • Nothing is unreasonable when you tell your customers they can pick any 2 items from the following list:

    You can have it fast.
    You can have it cheap.
    You can have it reliable.

    If they want it in 48 hours, you should explain what that would cost.

    If you want to streamline your process for patch deployment, I highly suggest you apply some six sigma or equivlant business process improvement strategy to it. Asking Slashdot is going to be counter productive. Your application is very different from anyone els

  • If a customer has encountered a serious issue and is unable to operate, getting a hot fix out ASAP to them is sensible. Turn around time might be in the region of 48 hours. By hot fix I mean fixing the specific issue only compared to the state of the source when the last release was shipped - aka the maintenance branch for the last release.

    In terms of period between each formal release; that will depend, but we try to have complete iterations (release or not) every three weeks. Ideally it should be two week
  • I would suspect this largely depends on the kind of application you're developing.

    Major problems in "cool blog software", for example, aren't really a huge issue. If it takes them a few weeks to be solved, then poor bloggers will be without some magic-pixie-dust feature for a bit.

    In the telecomm world, though, customers expect a root cause for a "critical" defect in less than 24 hours (and there's a definition for critical, although I won't get into that here). My company actually writes that into our sup
  • Two prong approach (Score:4, Informative)

    by khendron ( 225184 ) on Tuesday November 13, 2007 @05:39PM (#21342005) Homepage
    I've had situations with customers who require a fix as soon as possible, because if the system is down they are losing money. When this situation occurs, we have two goals in mind:

    (1) Get the customer up and running again as fast as possible. This is as often as not some sort of workaround that is not pretty, nor is it permanent, but it works. The workaround does get thorough testing (impossible within the time frame) but the customer is aware of this and willing to accept the risks.

    (2) Get the customer a proper, version controlled, patch that they can install to fix the problem permanently. This can take weeks, most of that time being testing. If the customer is insistent we will ship them the proper patch before it is fully tested (again, making them aware of the risks) and continue testing so that we can send the customer some warm and fuzzy news later on (or, if we find a problem, another patch).
  • by postbigbang ( 761081 ) on Tuesday November 13, 2007 @05:39PM (#21342009)
    Show stoppers get immediate attention; whatever it takes. People are losing money because they're DOWN. Fix it now.

    Criticals get next attention when show stoppers are out. 48 hours, depending on interdependencies and QA needed to make it work; it's not part of an official stable code tree until later.

    Minors are in the next stable branch release; every whatever you can handle.

    Nigglies are changed when the stable branch releases.

    Don't deviate from your policy, and make sure the sales people KNOW AND UNDERSTAND what this indicates and implies. No exceptions; see above.
  • It depends on the level of comfort you need to have with your release, and how quickly you can get there.
    I also work at a mid-sized company in the enterprise software business. We do a lot of automated testing. For an urgent, customer blocking issue, I could potentially:

    1) get really lucky and diagnose the problem instantaneously. Realistically, it will take me at least one hour to properly diagnose any serious bug that escaped our qa process.
    2) Make a fix. Assuming this is a fairly trivial fix, this c
  • I'm surprised nobody has mentioned the fastest responses, which may take essentially zero time. One is: "It isn't a bug.". The other is: "Already fixed. Update your installation."

  • by QuasiEvil ( 74356 ) on Tuesday November 13, 2007 @05:43PM (#21342053)
    I'm an embedded developer, and when my stuff goes wrong, it can *really* do bad stuff. I've literally pushed fixed firmware to a controller running in a production scan/sort environment within five minutes of finding the bug, because it threatened to completely bring down a huge sort operation (and by huge, I mean 1 million+ pieces that day alone). I've also stayed up all night tracking down a bug crashing a device used by one of our larger (hundreds of millions of dollars per year) customers. Those, though, are the exception, and are driven by the massive financial and PR consequences of not getting it done right now. Throw caution to the wind, code and load if you are reasonable sure what's wrong and the stakes of not fixing it are high enough.

    The usual bug fix cycle depends on complexity, impact, and risk. High risk of breaking things and low impact? Generally gets scheduled for the next release (4ish times per year). Low complexity and risk but medium impact? Code today, regression test the rest of the week, push this weekend. On average, mission critical bugs can get fixed in 8 hours or less around here, small to medium stuff is put on a weekly(ish) cycle with *lots* and *lots* of testing, and large stuff gets rolled to the next major release, unless it just can't wait that long.
  • I don't think there's enough info to even answer this one. I think the answer is entirely dependant on the importance of the program you put out. If you write rules for DoD firwalls, then you should be measuring in hours or minutes. If you write screensavers to put on OLPC... You have a little more time. And there's the inbetween, like if you write the algorithm that counts how many M&M's go into each bag.
  • Minimum turnaround.

    Large product. Billions in sales go through it.
    ITQA takes 10 days with no defect- 20 days with a defect.

    Scheduling installation time, Sox forms, required paperwork overhead, project approval comprise the rest. Actual coding and testing probably bout 8 to 16 hours.

    At a small privately held company, it would take us 1 to 3 days.
  • That's why there are companies who think a minor bug fix, or a small development, changing fonts or simple features, reconfiguring servers, restoring backups etc is something that doesn't need testing, concentration, at least little bit of planning and basic things like version control. So that's quite common in the industry: customers who think they are getting their product for less money because they can force every change as an emergency. They don't realize they are making development more expensive wit
  • by SystemFault ( 876435 ) on Tuesday November 13, 2007 @05:50PM (#21342165)
    From nearly forty years of programming (yes, since the IBM 026 keypunch days), I can tell you with absolute certainty that the more that you do for management, the more that they will want from you. It is not your responsibility to bear all the punishment for the lack of foresight and resource allocation on their part.

    Consider this: What would be the managerial response if you asked for a cost of living salary increase and that you needed it within 48 hours? Do you think that they would be willing to work day and night to make that happen?

    Working in panic mode is not professional behavior, and it certainly is not conductive to good engineering practices. Furthermore, it is detrimental to long term company survival. Engineers who support continued unreasonable demands have only themselves to blame for enabling poor strategic planning by management.

  • by PatMcGee ( 710105 ) on Tuesday November 13, 2007 @06:32PM (#21342705)
    The customer described a program they wanted (to run on an embedded system). I estimated 3-4 months. They asked for 30 days or less. I explained what they'd get if I banged it out that fast - something that would work most of the time and not lose too much data. They then explained that the program would save them over $1,000,000 a month. If it quit working, they quit saving money, but nothing else bad would happen.

    So, I saluted and said I'd try really hard for 3 weeks for the first version, then about three months longer for a version that would work all the time. Which is what happened.

    Do you know the impact on this customer of not having the fix that soon? Maybe it's worth it to them...
  • by JWSmythe ( 446288 ) * <> on Tuesday November 13, 2007 @07:06PM (#21343059) Homepage Journal

        Really, it depends on your environment, and what needs to be done.

        I'll use one of my web site as an example. It's all PHP and Perl, so ya, it's programming (I'm sure people will argue this).

        Since I wrote all the code, I know it all inside and out. If you say "there's a problem [here]", I know exactly what file to look in, and what code to look for. I've banged out changes, tested them, and put them into production in a matter of minutes.

        On a high traffic web site, we had a java applet which was being used by about 25,000 people per day. For little things, I'd change the code, test on all applicable platforms, and roll out the change in a few hours. Even then, the bosses were sometimes displeased with the time it took. Since I was careful to test, I never rolled out bad code, so I was never pushed into the long QA cycles.

        Working with one company, things were a lot different. It went something like this.

        1) Propose the change to your manager, with supporting documentation.
        2) Manager would go to the project coordinator (i.e., customer liaison)
        3) project coordinator would go to the customer
        4) customer would approve the change.

        Up to here was anywhere from an hour to a week. Sometimes the customer would put stipulations on the change, such as "there's a big event happening, or going to happen, don't make the change until X time."

        5) document the proposed changes
        6) hold a meeting with development, QA, the project coordinator, and management. Discuss the potential

        1-3 days later

        7) hold another meeting with the same people to rehash the changes.

        1-3 days later

        8) hold another meeting with the same people to rehash the changes.
        9) Write the changes. Make them available to the QA team.

        3-7 days later

        10) Explain to the QA team that the errors they are experiencing with the fix have nothing to do with the fix, they were preexisting problems with another piece of code.

        1-7 days later

        11) hold another meeting with development, QA, project coordinator, and management, to explain that the error has been fixed with the supplied changes. The other problems are elsewhere.

        1-3 days later

        12) hold a strategy meeting to plan on how to fix the other problems.
        13) fix the other problems, and break more things.

        1-3 days later

        14) have QA test the other changes.
        14) roll back changes in step 13

        15) beta test the previous changes, and notify customer
        16) Customer balks at other pre-existing problems.
        17) Repeat steps 5 to 15 again, until the customer gets tired of balking.
        18) Implement changes.

        Then start the process all over with step 1 to fix the other pre-existing problems.

        The solution really is...

        1) Identify the problem.
        2) Gather together the appropriate staff who won't talk outside of your group.
        3) Fix, internally test, and implement the resolution.
        4) If anyone asks, there was no problem to start with, and you were all really working on steps 5 to 15 of the previous plan on another problem.

        Funny how that works.

        But, it's a matter of, is it a trivial fix, or something that requires serious rewriting? Did someone miss trapping invalid input in one line, or is it a poor coding practice through all of the code? Is it an included library that simply needs to be upgraded and recompiled?

%DCL-MEM-BAD, bad memory VMS-F-PDGERS, pudding between the ears