Forgot your password?
typodupeerror
Cloud

Ask Slashdot: How Often Do You Push To Production? 182

Posted by samzenpus
from the hold-on-we're-going-live dept.
First time accepted submitter Stiletto writes "I work for a traditional 'old school' software company that is trying to move into web services, now competing with smaller, nimbler 'Web 2.0' companies. Unfortunately our release process is still stuck in the '90s. Paperwork and forms, sign-off meetings, and documentation approvals make it impossible to do even minor deployments to production faster than once a month. Major releases go out a couple of times a year. I've heard from colleagues in Bay Area companies who release weekly or daily (or even multiple times a day), allowing them to adapt quickly. Slashdotters, how often do you push software changes into production, and what best practices allow you to maintain that deployment rate without chaos?"
This discussion has been archived. No new comments can be posted.

Ask Slashdot: How Often Do You Push To Production?

Comments Filter:
  • by PieDkaj (2750665) on Thursday October 11, 2012 @02:03PM (#41621205)
    I push to production fairly often. Our company's Visual Studios have been configured with test cases that make testing easy, and therefore we can push to production much more often than those who do not use Visual Studio.
  • by Anonymous Coward on Thursday October 11, 2012 @02:06PM (#41621231)

    Why not talk to your colleagues and suggest ways of speeding things up?

    It's good to talk.

    Release when ready is quite a good frequency :-)

    • by Anonymous Coward on Thursday October 11, 2012 @02:19PM (#41621417)

      I think the submitter IS talking to colleagues and looking for suggestions right here...

      We do a major release about every 2 months, with maintenance stuff going every 2 weeks. There's pressure to speed this up. On the release side of things we're streamlining our documentation requirements and automating everything we can. We're also moving to a continuous delivery model for our regular test builds... if it's tagged, it goes to QA and an automagical report goes out with a list of resolved defects. This requires the developers to package things cleanly and consistently, but it also gives us an amazing amount of data to mine and I like to think that it helps to improve our overall quality.

      This doesn't address requirements gathering or development methodologies, but I can't really speak to those.

    • by Safety Cap (253500) on Thursday October 11, 2012 @02:30PM (#41621527) Homepage Journal

      If he can't deploy to production in one step, he needs to fix that first.

      I'm not talking about from dev box to production. I'm talking about the physical act of someone running a single command (or for the Winlazy, pressing a button) then walking away. All code checkouts from source control, database changes, app server code deployments, web server restarts—whatever—happen without user intervention.

      He should also be able to roll back in one step.

      For all the meetings, forms, etc., it sounds like there is A LOT of CYA in that company. In that case, it is cultural and can only be changed from the top. Until/unless the company becomes less risk adverse, there is no point in trying to become more "agile" (i.e., risk-accepting) except making your job easier. Build your tools/scripts/whatever to make it easier to do stuff.

      tl;dr: If you want a more nimble company, switch jobs.

      • Re: (Score:3, Informative)

        Absolutely

        One-step prep/audit/deploy/post-deploy audit/rollback/post rollback audit are critical and not dependent on your tool stack, pretty much any stack can be made to perform this function with proper attitudes/process design/automation. (also needs some break-point analysis ~ things like DB schema changes complicate roll-backs if you don't want to lose production data - some low latency design shops leave error checking out of certain high traffic code areas on purpose and they'll break on a rollback

        • Re: (Score:3, Insightful)

          Oh yes, and its worth noting that if you're not careful - easy deployment can quickly result in lower software quality. If you make deployment too easy quality drops like a rock if people think they can fix it easily the next day or even intraday. Some cultures reward by the # of releases/features in prod without looking at the total impact. 'Faster' is not necessarily better although useful in certain situations certainly.

      • by turgid (580780)

        there is no point in trying to become more "agile" (i.e., risk-accepting) except making your job easier.

        Agile is not about being more accepting of risk: it's about engineering risk out, reducing the scope for human error, testing early and often (with as much automation as possible) to catch bugs and other errors before they become expensive, being confident of not breaking things when things change and, above all, having as much confidence in the product as possible that it works.

        Everything else is a ver

        • Agile is not about being more accepting of risk: it's about engineering risk out,

          I actually wasn't talking about Agile the process but agile the adjective (hence the quotes). Perhaps I should've said 'flexible' or 'responsive'.

      • I'm talking about the physical act of someone running a single command (or for the Winlazy, pressing a button) then walking away.

        Walking away? From a release in progress? Do you worship at the Shrine of Murphy or something?

  • chaos (Score:4, Insightful)

    by Anonymous Coward on Thursday October 11, 2012 @02:08PM (#41621257)

    >> Slashdotters, how often do you push software changes into production,

    3 times a month, generally.

    >> and what best practices allow you to maintain that deployment rate without chaos?"

    IMHO, you can't. Having a deadline every week, programmers cut corners to make dates. QA cuts corners to make the date. This much code changing without any "bake time" inevitably leads to an unstable code base, full of "corner cuts" from the last N releases.

    • by jythie (914043)
      This pretty much sums up my thoughts. While there is always value in cutting away unnesseary beucracy, once that is gone you get into the tradeoff between flexibility and stability. Plenty of web-centric companies follow waterfall or other risk adverse patterns and do just fine.

      So it really comes down to examining how important rapid changes vs stability/validity are. Many of those 'web2.0' companies that can adjust rapidly are pretty flakey and have no paper trail to be able to show auditors (or underw
    • by Kjella (173770)

      IMHO, you can't. Having a deadline every week, programmers cut corners to make dates.

      More than if they risk missing the spring/fall release? Because that's really the question, deadlines always makes you cut corners but does fewer and bigger work better than fewer and smaller? I'm not so sure, I've seen people work on compliance projects where it's this one huge, absolute deadline and from what I hear it can get really hairy on the final stretch.

  • by knetcomp (1611179) on Thursday October 11, 2012 @02:10PM (#41621283)
    Most companies I've worked for have a continuous deployment cycle. All changes, from small bug fixes to major releases go through a ticketing system. After the ticket has gone through all the steps (code review, QA, UAT) it goes into the deployment manager's queue, who then deploys the change to production depending on each ticket's priority. This means that in general, changes go out as soon as they are ready, sometimes up to two times a day for the same project.
  • Set a schedule (Score:2, Insightful)

    by Anonymous Coward

    I think customers don't really NEED the frequent updates that you're getting used to seeing with apps. I would think only the foolish really expect rapid releases. That's been a side-effect of "apps" - not what we expect of software in the business world. If you set and can stick to a release schedule, I think your clients will appreciate it.

    End users find updating all the time a headache - only super-geeks like us like seeing every possible iteration of an application. The average user doesn't want to

    • Java and Flash updates are annoying because the user has to update. Compare with Chrome silent updates; do you really think the user cares if they're on version 13 or 24?

      I think small regular updates are better, since the user doesn't login one day, see 3/4 of the application changed, throw up their hands and assume they can't use it anymore.

      Small, incremental updates let the user take in changed they can handle.

      • by JohnFen (1641097)

        Java and Flash updates are annoying because the user has to update. Compare with Chrome silent updates; do you really think the user cares if they're on version 13 or 24?

        True, silent updates reduce one of the annoying things about rapid release, but it doesn't touch the rest of them. I stopped using Chrome in part because of rapid-release. I don't care what the Chrome version number is, but I care a great deal about unexpected changes.

      • by AuMatar (183847)

        The best is update when I say to. Unless its adding a feature that I personally will use, I don't want the UI to change AT ALL. When I'm ready for a new major version I'll learn it all at once. Until then, I want it static. That's the reason I'm using SeaMonkey and not Chrome or Firefox.

    • by JohnFen (1641097)

      I agree totally with this, but I may be biased. As a user, I find this rapid-release trend to be incredibly annoying. So annoying that it is a factor against my using a particular piece of software. Aside from emergencies, I cannot think of any compelling benefit rapid-release gives me, and it gives me two main headaches: an ever-changing piece of software, and the introduction of new bugs at any random time -- so it's a net minus.

  • Depends (Score:5, Insightful)

    by falcon5768 (629591) <Falcon5768@@@comcast...net> on Thursday October 11, 2012 @02:13PM (#41621337) Journal
    This really seems like it depends on WHAT your company does. Pushing to production a browser or a utility weekly or even daily is a lot easier and less disruptive than say a OS or a major software product thats used by hundreds of large companies. I would say the biggest obstacle is to get rid of the paper process, bring in git or something else for changes that everyone can view, and limit the signing offs to production, QA, or gold master (or whatever your company calls them) while keeping the dev stuff out of it so that the dev team can be more flexible. Also have a pretty good test lab/suite that will enable you to test stuff on the fly on multiple machines and configuration. Case in point Apple has a testing lab of EVERY machine they have made for the last 3 years in multiple configurations for their software. Granted they are a closed ecosystem so its easy for them to do that, but generally you could get a good roundup of machines to use that span from the insane configs to the "are you fing kidding me you are using that???" type stuff.
    • For us, it depends on the company's current beer budget. Forward the question to the CFO, please.

    • by Marillion (33728)

      I whole heartedly agree with this post.

      I'd like augment the remark about WHAT a company does. If your company is in the financial services sector in the US, then there are Sarbanes/Oxley regulations that must be satisfied. If your company is in the healthcare industry, there are privacy laws. If you're in the aviation industry, there's the FAA. And so on ...

      Also don't feel like you have to push out updates every week because it's the cool and trendy thing to do. Do it because there is a rational justif

    • by chrismcb (983081)

      This really seems like it depends on WHAT your company does.

      That is the key there. Our website is very content oriented. We release every week to 10 days. Mainly to push our new content out. But at the same time the newest features go along for the ride.
      If your product is more feature oriented, release when the features are done. Be feature driven, not time driven. There is no need to "release once a week" because someone else does it. Release when you hit your milestones. Release when the feature set is finished.

  • by John Hasler (414242) on Thursday October 11, 2012 @02:13PM (#41621349) Homepage

    Or were you looking forward to having to explain why your database got cracked and leaked 2 million passwords or that tens of thousands of customer machines will have to be manually patched to repair the damage done by your last update?

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      Wait, what? How about explaining why your database got cracked due to your *lack* of updates?

      You can do rapid deployments poorly and you can do them well. But when you really really need to deploy and *can't* because of the red tape involved, you're in bad shape.

      • by JohnFen (1641097)

        You can do rapid deployments poorly and you can do them well.

        True, as with all things.

        But when you really really need to deploy and *can't* because of the red tape involved, you're in bad shape.

        But if you're in that position, the problem isn't your choice of release model, it's that you're doing your release model poorly.

  • by Anonymous Coward on Thursday October 11, 2012 @02:15PM (#41621371)

    Sounds like yours wants a process. You push as the business feels comfortable with it.

    But think about this...

    Lets say you are dealing with peoples financial statements. You want to make sure they are pretty good and accurate. If it is wrong people loose money and you get sued. Your audience dislikes sudden change. So you put process around that to insure that (test/qa/forms/etc).

    Now lets say you are running a forum where it is the stats from some online game. It can be a little off. Your audience likes sudden new features. Stuff might not render correctly... But you can fix it tomorrow and the only gripe will be on a forum.

    One business you have to be careful in creating 0 downtime and 0 mistakes. The other, a mistake here and there is tolerated and can be fixed at will.

    Now I am not saying you can only create the first one where it takes a month or two for a push. I am saying those sorts of situations where caution is warranted. If you want a quicker process your process will need to be able to break things into small pieces and a bad ass testing system. Some situations they are contractually obligated to go at a particular rate. If you can not see the difference then I suggest you let your question simmer for a few years and get some exp doing it.

  • by Malenx (1453851) on Thursday October 11, 2012 @02:15PM (#41621373)

    Slashdot posted this story earlier,

    http://www.businessweek.com/articles/2012-10-04/facebook-the-making-of-1-billion-users [businessweek.com]

    Interesting read on how Facebook, arguably the largest website challenge so far this century updates daily.

  • by preaction (1526109) on Thursday October 11, 2012 @02:16PM (#41621385)

    At $dayjob, we decide to release and our process takes about an hour. All the automated tests are run by Jenkins CI, and are run again during the release on every box being deployed to in order to ensure stability. We tend to deploy to User-Acceptance Testing boxes before full production boxes.

    At the game company, we wrote a system that works like:

    1) Tag release in git
    2) Release is pushed to beta servers. Beta players get immediate updates.
    3) Click button in Jenkins to run stable release

    Completely automated, even down to restarting the servers in a staggered fashion to ensure that users always have a game to connect to (even if they have to disconnect in 20 minutes to receive the client update).

    Automated testing, Continual Integration, and automated release processes (including cfengine3 and custom Perl scripts working with Git) come together to produce a painless release process. Since it's easy, we can do it whenever we want. As soon as it starts releasing bad code, we'll have to put process in place to ensure bad code does not reach our stable users.

  • The goal is to always be able to push to production. Have a continuous integration and test system that allows you to have confidence that you can always push the latest build to production. Automate as much of your testing as possible so your build and test process can produce something that a human test team actually has to work hard to break.

    Also, you should pipeline your approval process. Always be in the process of approving a new build to go out while you're working on the next one. This will put a lot of pressure on this process to be faster and more efficient. Holding the people responsible for this process accountable for production breaks will put counter-pressure ensuring that they do not become more efficient at the expense of actually doing the job.

    • by cplusplus (782679)

      The goal is to always be able to push to production. Have a continuous integration and test system that allows you to have confidence that you can always push the latest build to production. Automate as much of your testing as possible so your build and test process can produce something that a human test team actually has to work hard to break.

      Parent hit the nail on the head. The above comment is the key to a successful agile development environment. Investing in a robust, automated test suite that runs daily on any code that developers check in will allow you to release about as frequently as you want.

  • Get rid of the clue less PHB's lot's of meetings is sign of being over loaded with PHB's.

  • by Glires (200409) on Thursday October 11, 2012 @02:17PM (#41621395)

    Sometimes all of those meetings and paperwork serve a useful purpose when an application is critical. If a one-day build of instagram doesn't work, then the only consequence is that there are fewer grainy photographs of someone's cat. If a one-day build of a power distribution system doesn't work, then an entire city loses electricity.

  • by DeadSea (69598) on Thursday October 11, 2012 @02:17PM (#41621401) Homepage Journal

    Push to production as soon as the (many) automated tests that you have pass. This means you should have comprehensive unit tests and tests that run in the browser, probably written in Selenium. You'll also want to script your release so that you can do it with the push of a button. Once the tests pass, and the mechanics of a release are trivial, there is little reason to hold up a release.

    I worked for a top 500 website (East coast) for 7 years that did weekly releases. Since I left, they decided that wasn't fast enough and now release multiple times per week. I'm now self-employed on my own website and release within an hour of finishing development of a feature.

    I started my development career writing firmware for laser printers. When you are shipping code on a physical product, the cost of bugs can be quite high. Especially when it leads to returns or recalls because customers are not satisfied. Our release cycles there were 6 months+. Quite appropriately, IMO.

    On the web, the cost of bugs is much lower. In most cases it is the only cost of another release. Sometimes it could cost more because of downtime, but good automated test coverage mitigates that risk pretty well (especially if there is load testing involved). The worst case would be data-corruption, but I've never actually seen that in practice from a release, that has only been related to hardware failure or accidents in my experience.

  • What you need to be asking is, "How long is an acceptable test cycle?". Considering something "in production" usually implies some sort of testing, otherwise all the changes would just show up immediately right? So, ask yourself (or your organization) how much testing is enough? Equally important, how much is too much? These are the questions every software company struggles with. Spend too much time testing and your releases will always be behind. Spend too little time on it, and your "production" wil

  • I work at a company with many brands and MANY customer-facing websites. Some sites/services can be updated in minutes with little hassle or approval. Some need weeks with much paperwork. It depends on the risk. The apps are built in different ways (CMS or from scratch or with many proven components). The apps have different features (some can result in $million fines if there's a problem).

    The tools are there to set up processes where every time I commit, it goes to production. You need a process that

  • As a customer (Score:5, Insightful)

    by ZombieBraintrust (1685608) on Thursday October 11, 2012 @02:22PM (#41621445)
    As a customer I hate daily updates to my applications. Unless the application is in Alpha or Beta it is very disconcerting to see updates at that frequency. I only want to see an update that quick when something is really broken. If brokeness is a daily occurence then maybe you need to slow down.
  • by cryfreedomlove (929828) on Thursday October 11, 2012 @02:23PM (#41621449)
    Life is short. You seem like a talented person who knows what you want. If you current company does not modernize in the next 6 months then they are holding you back. The market is starved for really talented software engineers. Take the time to research your opportunities and then jump ship if the one you are on is still stuck in the 90's.
    • Life is short. You seem like a talented person who knows what you want ... the market is starved for really talented software engineers ...

      Amen to that, brothers and sisters! Amen to that!

  • each way the release is tested and signed off on. little things are tested like making sure that the financial transactions are OK and customers are overcharged or undercharged. One time we had a case where we undercharged customers for years, then sent them a bill for a few years services and ended up in a lawsuit.

    other times code works on developer laptops but somehow it doesn't work in production

    i'm at the point where unless you're facebook or someone big i never report issues. i just use your competitio

  • I sometimes just work in the live environment while people are in and using it, depending on what I'm doing.. I'm very careful when I'm doing this of course, but sometimes the hotfix requires having the user's screwed up data to work with.

    The rest of the time, I publish when I think I have something useful for the users.. Often a couple of times a day, but we're one of those very small nimble web 2.0 places, and the decision to publish is all mine.

  • Pushing to production multiple times a day sounds like code-and-fix to me. Now, promotion to UAT/Test multiple times a day sounds much more reasonable. Pushes to production should be regular and frequent but not multiple times a day. In my company, we have a couple of change windows that production pushes happen in. Most times the windows come and go with no updates but when we do have code that's ready for production we schedule it for the next available window.
  • Path to Victory (Score:4, Interesting)

    by Electrawn (321224) <electrawn@yahoo . c om> on Thursday October 11, 2012 @02:30PM (#41621525) Homepage

    I am a Release manager at Acquity Group and have worked for "Old School" software companies that have their eyes blazing at all the new web companies that release release release all day long.

    Here is the abbreviated philosophic path to victory:

    DevOps.

    Your developers need to act like operations (knowing how the code is deployed and configuration settings, routes and the like) guys and your operations guys need to start coding (as in ruby for puppet and chef, automation and automation and automation)

    This leads to...

    Infrastructure as Code.

    Hire a Release Engineer or convert a sysadmin to start automating builds. Now you start automating code deployments, you start automating infrastructure deployments so they are repeatable.

    This leads to...

    Test Automation.

    Now you need to stop focusing on smoke testing and have test automation engineers write automated test code.

    One more thing...

    Automated Rollback

    When things go nuts, with fast deploys you need fast rollback. Capistrano is a great tool for deploying this way, rollback is very easy.

    Now you can ...

    Continuous Delivery.

    Great, you got this far, your builds, testing, deploys are automated!

    For Developers:

    Coding for Continuous Delivery is a different paradigm where unfinished code makes it to production. This means that in the production configuration, settings for the new code must be activated by switches to turn new feature sets on. You don't want that unfinished code mucking up your site, right?

    People Processes:

    Do you have CAB boards and ITIL processes? Great, make them faster and more as DEV/TEST/QA becomes automated and just focus on UAT/Prod environments. See this book: http://www.amazon.com/The-Visible-Ops-Handbook-Implementing/dp/0975568612/ [amazon.com]

    I can also gloss over on waterfall/Agile and hybrid software models.

    Finally, unless your culture wants to shift, it may be damned near impossible to change the culture unless management wants to. If it's doomed, it's doomed!

  • Without any idea of what the product does, this isn't a meaningful question. Is this just a web site? A program used by people? A program used by other programs? How much persistent state is there? Do users need to know in advance before an update is made?

  • We only have a production environment to develop code, and use an interpreted language, so it's as simple as editing a file and sending a few HUP commands.

    Cardninal Rule: Don't fuck up.

    It's worked mostly well for years now.

  • On my recent project, we aim to deliver only once the software. Off course you always have to do some small modifications after the first productions test. In my last company after the launch of the production, I remember of 4 different push in production in one year. But it was a brand new product. (I have left the company since that time, but as far as I know there was no new modification in the production. A new feature should be pushed in a near future)

    In my current company, in one year I have not se

  • GitHub as an example (Score:4, Informative)

    by kav2k (1545689) on Thursday October 11, 2012 @02:36PM (#41621593)

    I invite you to read the GitHub blog post on how they deploy [github.com].

  • The company I work for is right smack in the middle of this transition. We had something akin to a SaaS model, before SaaS was a "thing". We have 40+ applications, some are desktop thick net .Net clients, others are web based, all of which interact with one another to varying degrees. Myself and one other person were instrumental in getting the company to a point where it is possible to release in a semi-automated fashion. Our challenges were similar to what you described - manual work with lots of proces
  • Every Wednesday is Preprod day, and every Sunday is Production Deploy. Production was done by a rotating group of about a dozen people, so you didn't have to work on Sunday too often.

    It was a rare week that had nothing to deploy. It was also rare (thank Fudd) for us to have to do an emergency backout on Monday.

    Speaking of which, it's good to have different teams testing and deploying. It's less likely that the deploy team will try to fix the bits after they're deployed in Prod, and maybe forget to fold t

  • by einar2 (784078) on Thursday October 11, 2012 @02:39PM (#41621629)
    It depends on your business which metric is meaningful. E.g. for a global bank, quality is more important than time to market. Make sure that your business really gains something by playing release time against quality.
    • by Kjella (173770)

      It depends on your business which metric is meaningful. E.g. for a global bank, quality is more important than time to market. Make sure that your business really gains something by playing release time against quality.

      Actually, I was at a seminar earlier today and that's really not true. Of course the core systems for your bank balance and online bank must have very high quality, but analysts, sales, marketing and management only care about big numbers, trends and probabilities and there's very much information that just has to be mostly right and that's all the information they need going forwards, how much money people have in the bank right now or used to have is of course important to running the business but most th

    • by Mabhatter (126906)

      For things like banking, compliance is the bigger issue most of the time. Bugs calculating somebody's checkbook are rare. But having new tax rules on tap Jan. 1 becomes most important.

      I work in MRP, good ole manufacturing. So the defining factor is SHIPPING and number two is GET PAID. Those can drive SAME DAY changes.

  • Monthly release cycle isn't really bad, if your goal is stability. This gives your testing/QA team (you have one, right?) plenty of time to find problems, and developers time to fix any bugs the QA team found.

    Shorter release cycles might be better if your goal is responsiveness, but in my experience faster than weekly is asking for trouble.

    Also, make sure you have a process for handling those things that do legitimately need to move faster than the release cycle (e.g. "We just discovered a massive bug that'

  • by fotbr (855184)

    We'd like to push stuff to production faster, but our client usually only has one window a year when we can take production systems offline for upgrades.

    As for keeping it organized...hah. Two weeks of pure chaos while the hardware folks are trying to get all their upgrades and replacements in place at the same time the software guys are trying to figure out when systems will be back together so they can do their upgrades, followed by a weekend of praying that everything comes back online the way it should.

  • I push to production fairly often, depending on the nature of the tickets I'm responding to. I'm generally maintaining three branches concurrently, the production branch, the point-release branch and the major release branch. Trivial fixes or small-but-critical fixes tend to go onto the production branch and get released as hot fixes. This can happen several times a day or not for several days, depending on how extensive the change is and thus how much testing is needed.

    Fixes that are a bit more extensive a

  • There is no one frequency that is good for every application- it depends far more on what you're doing and you should be focused on delivering that, rather than a pretty arbitrary metric. Look at the value of what you're doing. What is the cost of things breaking? Don't forget, MTBF *includes* the time to fix the failure.

    Think about live broadcasting. If you're selling informing people of what is happening right now, you better be able to recover from a link down quickly, even if it means not keeping rec

  • This shirt covers the methodology I've sadly seen far too often:

            http://www.thinkgeek.com/product/f141/ [thinkgeek.com] [Most Interesting Coder]

  • The answer you are looking for is: Lower company standards. Having worked for companies large and small, including start-ups at various stages, I can say from experience that frequent releases work because management has reasonably low standards, and is perfectly okay with the last deployment breaking things, and needing to be hot-fixed by a follow-up deployment just days later. That's just the model.

    I know the model you're talking about, too. A more solid company, building its web products like it woul

  • As often as we need. We push based on features. Although we sometimes slip into batching up production moves. I find it is much easier to push often when you work with people that are unafraid of branching based on feature requests.

    I'm just glad we don't do the "quarterly push" like some do. It sure saves us a lot of troubleshooting in the chance that the push introduces a production error.

  • I work for a smallish team (~100) in a megacorp that ships a service that has a significant number of customers.

    We have 2 major code branches - the developer branch and the production branch.
    - The production branch sees management approved point fixes that get collected and rolled out every month after a huge number of tests (automated and manual).
    - The developer branch gets continuous feature additions. We collect all features from the developer branch and move it to production once every 6 months.

    The deta

  • Well, whatever you do, please don't follow Google's lead, at least with respect to Google Groups.

    They seem to push a new version at least weekly, and sometimes daily.

    It's a coin-toss whether it will actually work from one day to the next. Features will just stop working for a few days, until somebody notices and fixes. Or fixes the fix that was put in to fix the fix.

    Latest is that Firefox 9.0 is "obsolete". If you are running anything older than a fresh-out-of-the-box (or Torrent) Linux release, you are pro

  • At one prior job, it was official written policy to NEVER upgrade to the latest release of ANYTHING, on the grounds that it had no track record. So it doesn't matter how often you release.

    Releasing on a fixed time schedule has never made sense to me. Target, yes, but if it's ready a week early, or needs a week longer, better to do it right than to do it at a fixed time.
    • by JohnFen (1641097)

      At one prior job, it was official written policy to NEVER upgrade to the latest release of ANYTHING

      That is a very, very wise policy.

      • Until the release schedule is so slow the version you're on is EOL'd and doesn't support the feature you need, but you can't get approval to upgrade

  • You're not eating the cost of burning releases to media.
    Push early, push often, but have a rock solid, bulletproof rollback mechanism.

    Seriously, that's what staging is for. push it, roll it back, push it, roll it back.
    Since you don't have to stick to a golden master, media reproduction schedule,
    push a feature as soon as it is ready.

    If your organization can't get past big quarterly releases, then they will not succeed with more nimble competition.
    The organization has to adapt. So wither the management adapts

  • At Verkkokauppa.com [verkkokauppa.com] we deploy several times a day. Our team is fortunate enough to work on our own system, so for small features or bug fixes the turnaround can be as little as minutes.

    For us the most important tool is Git. We have a logging mechanism in our deployments scripts whish logs who deployed which version of the code base (SHA1 hash) and when. If we do manage to break something we simply immediately return to the last known good version by checking it out and deploying it. Then we have a lockout p

    • by chrismcb (983081)

      At ... we deploy several times a day

      Why? Why is what you did at 11am SO important it has to get out right this minute, and can't wait till 5:00pm, or tomorrow or the end of the week?
      Why churn your feature set several times a day? What is so important it can't wait for a few days?
      Deploying that often is just a waste, and is unnecessary for most products

  • At netflix, each app owner (either person or group) figures out what release process and frequency make sense for them. Some apps get pushed extremely rarely -- probably for actively developed systems, the least frequent I see around here is every other week. Other apps get pushed more quickly. Lots of people who push weekly; lots of people who push ad-hoc, whenever there's a change they want to get into production. Our Engineering Tools people are also finally starting to support continuous deployment,

  • I work at an online email marketing company of about 50 people.

    We have weekly hotfixes for low/medium impact bugs. We can update all servers in 10 minutes for high impact bugs. We have a feature upgrade release every 6 to 9 months. We could release earlier but we don't want too many communication moments with our customers. The second reason is that we want to deliver a finished product which takes some time, documentation and marketing must be changed, etc.

    We don't communicate our maintenance schedu
  • We have a particular software from a particular vendor who only pushes an update once a year. We've already pointed out several bugs and quirks in the software that have become "feature requests" for the next release (due in January 2013). On the other hand, we also have software from a vendor who pushes an update every six weeks, and then we have to get that update installed for every user that has the software because every update changes the file format and it isn't ever backward compatible! . So, if you
  • And don't want to do it myself.
  • Unless you take into account the business you work for and the environment you work in then you've asked an incomplete question.

    In a small-ish company you can push out updates quicker because in many ways there's less risk: if there's a problem it either (a) doesn't impact as many people or (b) is easy to roll back. That's a generalization for sure and everyone can pull out exceptions I'd bet, but I think it's a fair generalization.

    In a large corporation though, especially one where there may be government

  • For my company, it's every 7 weeks with an 11-week lead time ... but it's going to be different for everyone depending on their business requirements. For my company, we must QA/Beta every development release before it makes it to production for general availability. To this end, I have produced a schedule:

    * 3 week scrum
    * 2 week first-pass QA
    * 2 week second-pass QA w/customer Beta
    *** dev team starts the next scrum cycle here, overlapping with the 4-week marketing period that follows
    * 4 week marketing
  • I work at a game company and we do about once a week, once every 24 hours for hotfixes. We can't really do sooner for several reasons but the foremost is that players want to play our games and they can't do that if we're bringing the servers down. We use some heavy automation (and always getting heavier. Currently transitioning from in-house + bash scripts to bash scripts + git + Puppet) which acts as a force multiplier for your teams.

    However, PLEASE be careful when doing automation and sending build
  • I've found that for web development today, weekly or twice a week is around about the sweet spot for non-trivial web-development.
    Of course you need a modern pipeline. Good devtools, a framework, test first and a lead architect and scrum master who knows what he's doing. But the days of paper driven management in software development are totally over, except maybe for spaceships, military hardware, nuclear power plants and perhaps medical gear.

    If you're doing web-dev management with a manually maintained pap

  • Just leaving that here....

    You just code the changes in your own library, then copy to the production library.... Or just edit in production, because the program object doesn't update until you compile.

  • Mind you I am the sole IT employee of a small company, so the entire process is under my control. FWIW It's a data-driven B2B web app kind of thing.

    I prefer to split up change requests - including configuration changes - into individual numbered releases, which are tagged in SVN and documented on the login page visible to internal users, so the evolution of the app - and my contribution - is visible, especially for the people who wouldn't otherwise see what I'm doing.

  • I find "Adapt Quickly" is a metaphor for "insufficiently tested" or "it worked on my machine"

    If I had a dollar for every "tiny fix" that required hours of hacking in prod to resolve with the release team sitting around waiting for the wunderkind to figure it out, for it then to be reverted and done again the next weekend. So much better when properly tested with signoffs and accountability.

  • We usually make several production releases every day. We have a complete configuration management system (conform) that totally automates building a server and releasing software. We have a complete dev environment and two test environments. We test code in pre-production first. But the key is our automation. It automates releases and rollbacks. Without complete automation (and competence), we would be submerged in paper work and bleary-eyed midnight releases like other teams in the company.

FORTUNE'S FUN FACTS TO KNOW AND TELL: #44 Zebras are colored with dark stripes on a light background.

Working...