Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
Businesses Programming IT

Who Owns Deployments - Dev or IT? 152

Posted by Cliff from the custody-battles-are-always-nasty dept.
txpenguin asks: "I am IT manager for a small software company. We host several generations of our applications in a fairly complex environment. Our systems are very much inter-dependent (clustering, replication, heavily loaded, and so forth), and bad changes tend to have a domino effect. Additionally, it seems that there are always those who need to be 'in the loop', but aren't aware of changes which affect them. There is a constant battle between IT and Development regarding who should handle the deployment of new code releases and database schema changes to production systems. Dev doesn't understand the systems, and IT does not know the code well. How do you handle this at your company? What protocols seem to work best? Can there be a middle ground?"
This discussion has been archived. No new comments can be posted.

Who Owns Deployments - Dev or IT? More

Who Owns Deployments - Dev or IT?

Comments Filter:

  • Middle ground (Score:5, Insightful)

    by Southpaw018 ( 793465 ) * on Wednesday December 13, 2006 @07:49AM (#17221036) Journal
    These kinds of things where there are two opposing sides always have the same answer. Unless one side is teh debil or something.

    You have to compromise. That's it. Middle ground. There are no other solutions to or ways around this problem. As you describe it, each side has access to and knowledge of half the problem. Half plus half is whole!

    So, meet with the guys in Dev. If you want to be beaureaucratic and official about it, create a "deployment team" consisting of an equal number of members from each side that will sit down, discuss, and supervise all necessary changes to production systems. Hell, send someone to a project management class if you need to.

    Now, the obstacle you're likely to hit is office politics. People won't want to listen to others and/or won't want to give up their turf or allow others on it. Too bad. To place how serious this issue is in overcoming the political terms: everyone in both departments needs to be cooperating or unemployed.

    So there you go. Just like any other relationship, business or otherwise: sit down and talk it over. Problems solved!

    • Re:Middle ground (Score:4, Insightful)

      by popeyethesailor ( 325796 ) on Wednesday December 13, 2006 @08:13AM (#17221152)
      There's no need for a compromise.
      Developers write code & documentation.
      Installation/Deployment guides *are* required documentation.
      IT takes the software, and follows the guide.

      Business applications are mostly consolidated on a few servers. IT guys know dependencies, time windows, batch runs and the whole shebang. A dev team has no business doing all this.

      • Re:Middle ground (Score:5, Funny)

        by Schraegstrichpunkt ( 931443 ) on Wednesday December 13, 2006 @08:24AM (#17221232) Homepage

        There's no need for a compromise.
        Developers write code & documentation.
        Installation/Deployment guides *are* required documentation.
        IT takes the software, and follows the guide.

        And in Magic Happy Land, that actually works without a problem.


        • And in Magic Happy Land, that actually works without a problem.

          But it doesnt have to. That's why in a good shop, there's actually a process for rolling-back the install, contingency plans, business continuity and disaster recovery. It's not a trivial process; even more important that there's a clear separation of responsibilities.

          • Re: (Score:3, Insightful)

            by ComputerizedYoga ( 466024 )
            err, then there's also testing.

            "rolling back the install" is a low-grade disaster recovery scenario. Testing the install on a non-production machine, working out the install/upgrade kinks and maybe even having a team of testers or some scripted testcases to throw at it before you start doing anything on the production systems is disaster prevention.

            And any doctor, sysadmin, or person with a modicum of common sense (or at least familiarity with some common-sense aphorisms) will tell you something about the
        • It may sound highly bureaucratic to you, and yes, for a small change is like 5x times more slow but it works if you establish some policies.

          I work for a fairly sized enterprise. There are 20+ engineers in development and only 6 of us in "infrastructure" (sys admins). Development teams simply don't touch the Test or Production environments, they have their own domain and servers which we help them to run on a you-broke-it-you-fix-it policy. All code, db scripts, catalog data, etc is managed by a Change Manag

        • Re: (Score:3, Insightful)

          by hondo77 ( 324058 )

          And in Magic Happy Land, that actually works without a problem.

          That land is called Sarbanes-Oxley Land and it has to work or you fail your audit.

          • Re: (Score:3, Funny)

            by russotto ( 537200 )
            Yep, because a law mandating something always means that something will happen.

            • Re: (Score:2)

              by pyite ( 140350 )
              It works in companies where minutes of downtime translates to tens of thousands of dollars. Yes, the pressure is high, but the rewards are great and it's also nice working with people who actually care.

          • Re: (Score:2)

            by Sux2BU ( 20893 )
            Sarbanes-Oxley...it works because it has to!

      • Re:Middle ground (Score:4, Informative)

        by mjpaci ( 33725 ) * on Wednesday December 13, 2006 @09:53AM (#17222048) Homepage Journal
        My company (~2,500 ppl) has moved to this model. However, we have dedicated "deployment engineers" who take the developers' documentation and follow it to a T. If there is an issue, they roll back. At first nobody liked the idea, but as time progressed it has started to work well. The deployment engineers were taken from the server teams and the DBA team and formed into a new group of about 10 people.

        --Mike

      • Re: (Score:3, Insightful)

        by Doctor Memory ( 6336 )

        Installation/Deployment guides *are* required documentation.
        IT takes the software, and follows the guide.

        And note that these are guides, not step-by-step instructions. They should say things like "Load the database schema update script (app_schema_updt_1_1_19.sql) into the database". The actual mechanics of doing this (making a backup, bringing an additional transaction log file on-line, starting table-level auditing, whatever) are left to the people who actually know the system (and the department procedures). Part of the development process is circulating a draft of the installation guide to IT for comme

      • For a one-installation in house application, building a shrinkwrap-strength administrator's guide and installation system is a big expense. Better to have high bandwidth communication between IT and development.

    • Re:Middle ground (Score:4, Insightful)

      by gbjbaanb ( 229885 ) on Wednesday December 13, 2006 @10:12AM (#17222250)
      Neither. You do not have to compromise if you're the boss and you require stuff to work. Office politics and willy-waving over who's more important should be a secondary issue to making the stuff work.

      So. you have a certification team (or quality team, or test team) who's job it is to certify that what dev has goven them works as dev said. These guys install it on their own separate systems that mirror the business (on a smaller scale) and test it out. Bugs get reported back to dev who get to fix them and so on. Eventually it'll get rolled out to IT who will have a reasonably good expectation that it'll all work.

      However - even in the best of cases there will be exceptional circumstances, and its at this point that IT will get dev members to come and fix up issue that arise on the live system. IT should be first contacting the cert team who will determine the bug (hopefully with a bit more inside knowledge to reproduce it on their systems) and will then get dev to issue a patch, which goes through the standard release process.

      Of course, if you want to let dev team hack about (which is probably why you have such a complex system in the first place), and IT to twiddle with their setup then fine - expect it all to go arse-up.

      I like to think of these environments as always having a 'customer' that they deliver to. If they provide a poor service, the customer has every right to complain. So, Dev's customer is the IT guys, IT's customer is the Business, and Business answers to real, paying customers. Such a chain of responsibility does focus people's attention on what they are trying to achieve for the company.

    • What about QA? (Score:4, Insightful)

      by trcooper ( 18794 ) * <coop@MONETredout.org minus painter> on Wednesday December 13, 2006 @10:41AM (#17222580) Homepage
      In my company QA is the bridge between development and production. I'm a team lead (dev) in a company which has a suite of web applications. Each application has a lead assigned to it, who handles the development and documentation of a product through their team. We do several deployments of software each week, and if our leads had to hand-hold through each of them we'd be hamstrung for time and working more night hours than we'd like.

      When we have a RC I'll branch the trunk, and request that QA perform a Pre-Production build. Developers will work with ops to get this running properly on the pre hardware, as this can be done outside of maint hours. We'll then do several builds of the branch until it's gold, and then tag off the branch as X.vv.zz.

      While a major release is in QA the lead focuses on creating/updating the operations document which addresses day-to-day maintainence issues and tells operations how to troubleshoot the app in the case of a problem. They also produce an implementation plan which identifies the groups/persons needed to deploy the application, and the steps needed to be taken, using what they've learned from the initial pre deployment. Once this is done, and QA has promoted the app, a dry-run is performed to try to catch any missing steps. The implementation plan is handed to QA, who coordinates with IT/Ops to resolve any conflicts and schedule the deployment. Ops/DBA's then physically performs the deployment following the steps given in the plan. In a major release situation, you may have a team lead or platform manager coordinating the actual steps on a conference bridge. But for minor releases we've been able to just have our operations teams do the full deployment with verification by QA and the product's customer service group.

      We also have a twice weekly meeting where any upcoming production changes are discussed between IT/Ops, QA and Dev. Release documents are put on a calendar, so if an issue comes up on another product we can go to this and see what may have caused it.

      Dev and QA also meet weekly to discuss the progression of products through or into QA. Any issues with testing or problems with builds not being stable can be addressed.

      It took us a while to get to this point. We had previously been in a situation where dev would handle the build and deployment process, and it was had for many of the leads to let their projects go, but now we can see the benefit, not only for the company, but also in the fact we don't have to be doing releases at 12AM on tuesdays anymore. It takes a lot of work across departments, and definitely is a long road, but something that needs to be done.

    • Re: (Score:2)

      by Fastolfe ( 1470 )

      Now, the obstacle you're likely to hit is office politics. People won't want to listen to others and/or won't want to give up their turf or allow others on it.

      I work in a large IT organization, and my experience is similarly negative, but exactly the opposite: Production Support wants nothing more than to have zero liability and zero work. Consequently, they think of every possible way they can push responsibility onto the development teams, including giving access to production servers. Stability isn'

    • These kinds of things where there are two opposing sides always have the same answer. Unless one side is teh debil or something.

      You have to compromise. That's it. Middle ground. There are no other solutions to or ways around this problem. As you describe it, each side has access to and knowledge of half the problem. Half plus half is whole!

      So, meet with the guys in Dev. If you want to be beaureaucratic and official about it, create a "deployment team" consisting of an equal number of members from each side

      • Be careful about this. I did something similar at my last job and I ended up getting slapped down pretty hard for it. The back channels are very useful for those of us who make/need them, but for the higher ups in management it can be seen as undermining their authority.

        Just to give you an example. I was working in a NOC when I noticed that a Tier 1 agent was making some mistakes. Nothing huge, and nothing that should have gotten them in any trouble, just stuff that was done mostly out of ignorance. Of

        • Be careful about this. I did something similar at my last job and I ended up getting slapped down pretty hard for it. The back channels are very useful for those of us who make/need them, but for the higher ups in management it can be seen as undermining their authority.

          Good point! Thankfully, all my back channel networks have been sanctioned by my superiors. I typically explain my intentions and motivations, etc., and it all works out. If the office politics are so bad that you aren't even allowed to wo

    • Sure, you compromise and you have people from both teams, but someone has to lead the show. It's a bad idea to not have a project lead. Project leads make sure someone is ultimately responsible for meeting deadlines, standards, etc.

      One project lead has to be from IT. The IT department is responsible for timing, security standards, rollback plan, etc. Then you have a project lead from development. The Dev lead is responsible for making sure the software actually works, but has no responsibility for prop
  • IT should only be involved in the maintenance of your company's network. The developers need to own the entire product, and if that means bringing in some database experts who can handle deployments, then that's a must. If you are a consumer of your own product (and a lot of companies do that as a way to be one of their own wins) then IT should handle the network right up to the point where your network starts hitting the product network.

    Of course you will need to have separated the networks in order for th

    • Re: (Score:3, Informative)

      by MyLongNickName ( 822545 )
      The developers need to own the entire product,

      So what about someone like me in the banking industry? You have any idea how long and hard the auditors will bitch if there isn't a separation of duties? Technically the developers are not to have ANY access to the production environment above that of an end user. (operative word being technically)
      • I work in banking as well, and here, the developers are not allowed ANY access to the production environment, period.
        An end user is an officer of the bank that is trained and authorised to transfer real money to/from customer accounts. No IT-developer has the right to even view the customer account balances. Some from IT-production technically can look, but Internal security guys do try to log and audit the records that they look at, and viewing a politician's/musicstar's bank account is con
    • The developers need to own the entire product

      No, whoever supports it and interacts with the customers should own it. If it is an internal product then that may be IT, if it is a product for outsiders then that may be Tech Support. I've worked in both environments, and while my ego as a developer says I am at the top of the food chain and should be in charge of everything, I realize that things worked pretty well when others owned the deployment and interacted with customers. It was an interesting enviro

    • Re: (Score:2)

      by dircha ( 893383 )
      "IT should only be involved in the maintenance of your company's network. The developers need to own the entire product, and if that means bringing in some database experts who can handle deployments, then that's a must."

      Absolutely not! Been there, done that. Development needs to be doing development. If Development are the only ones who know how to deploy, manage, and upgrade apps, then that is what they will spend much of their time doing. And because any failure in a production app is an immediate emerge

  • Speaking as a developer (Score:5, Insightful)

    by mwvdlee ( 775178 ) on Wednesday December 13, 2006 @07:58AM (#17221080) Homepage
    IT should own the deployments.
    Assuming the dev department does their job well, a deployment would not require any of the dev department's skills.

    • Re: (Score:3, Informative)

      by arachnoprobe ( 945081 )
      I think thats a good comment. That would also force the Dev Team to write a good documentation and spec.
      • That would also force the Dev Team to write a good documentation and spec.

        This works in theory. In practice, Management / the customer will be asking why the project is taking 3 months longer than planned - IT says Dev hasn't completed the docs. Dev will say the system is done and IT is just whining. Management dictates that the software get installed anyway, and tell Dev to get you the docs soon (don't hold your breath.)

        Been there, done that.

        It comes down to Money. If company income is on the line, best pr
    • That assumes that there aren't any quirks in the IT departments' design of the company network, systems, or policies, that any quirks are already documented, that the dev's system doesn't have any bugs, and that the dev's testing environment is completely equivalent to the production environment.

      Short of all that, they're going to need to work together on something.
      • That assumes that there aren't any quirks in the IT departments' design of the company network, systems, or policies, that any quirks are already documented, that the dev's system doesn't have any bugs, and that the dev's testing environment is completely equivalent to the production environment.

        This is why we have something called a QA department

    • Re:Speaking as a developer (Score:5, Interesting)

      by zrq ( 794138 ) on Wednesday December 13, 2006 @08:17AM (#17221188) Journal
      IT should own the deployments.
      Assuming the dev department does their job well, a deployment would not require any of the dev department's skills.

      Absolutely agree.
      If the developers do their job correctly, then a release should include a full set of install and migration instructions for IT to use.

      If IT do their job correctly, they should test the install on a separate system before deploying it live.
      If the install does not work 100% first time, don't tweak it, send it back.

      If the developers complain that IT didn't follow the instructions correctly, then the instructions were wrong.
      Send it back to the developers to write better instructions.

      If all goes pear shaped on the live system, IT should have a full set of (tested) instructions on how to rebuild the system from scratch.
      If the developers can't supply those instructions, then you don't know what you have.

      Ok, I know this is nice in theory and difficult to acheive in practice, but both teams should be working towards this as their goal.

      • Mod Parent Up (Score:2, Interesting)

        by A. Lynch ( 17937 )
        This scenario has always avoided the "pissing match" that inevitably occurs between Dev and IT, in my opinion. Clearly defined roles.

        In a previous shop, we treated our in-house developer code releases like any other vendor release. Just because some code was written by the guy down the hall, shouldn't mean that I can't ask for the same level of documentation/support I get from another Tier 1 vendor.

      • Re:Speaking as a developer (Score:4, Insightful)

        by SomeoneGotMyNick ( 200685 ) on Wednesday December 13, 2006 @08:40AM (#17221356) Journal
        If the developers complain that IT didn't follow the instructions correctly, then the instructions were wrong.
        Send it back to the developers to write better instructions.

        That's not always true, Mr. Nick Burns [wikipedia.org]. Sometimes IT has a permanent bug up their network port and refuses to learn a small amount of Developer's vernacular to share in the process. Likewise, Developers should not have to know how to speak 100% "IT" to write instructions. There is a common ground. IT personnel are paid for their experience, and ability to adapt, and not to simply follow instructions.
      • I agree, except I would put QA as a separate team between dev and IT. IT should not be allowed to deploy anything that QA has not approved. If they do, IT is at fault for whatever happens. By the time a deployment gets to IT, QA has verified that it works correctly. If it doesn't, and they pass it, then QA is at fault. If it should work but it's deployed wrong, then IT is at fault, because QA was able to follow the dev's instructions. If it doesn't pass QA, it's the dev's fault. This minimizes finger

      • Re: (Score:3, Funny)

        by jafac ( 1449 )
        What is this "test system" of which you speak?

        I keep hearing people talk about "non-production hardware" - and it sounds like gibberish.

        I asked my manager about this, and he snapped the pencil he was holding and started muttering something about "damn commies.". . .

        Yes - there is a "Fantasy World" - it's where people are promoted into management for competence, and allocate sufficient resources for projects, and blue fairies ride on rainbows throwing bags of candy.

        In the real world, we "test" - hell, we DE

        • Re: (Score:2)

          by larien ( 5608 )
          Sounds like someone should figure out that cost of test/dev system (cost of downtime because you couldn't test) * (probability of downtime). If, on the other hand, the cost of downtime is less than the dev system, that needs to be highlighted as a potential risk. If the company can't figure that out, they deserve everything they get.

          • Re: (Score:2)

            by jafac ( 1449 )
            If the company can't figure that out, they deserve everything they get.

            . . . million-dollar defense contracts. . .

      • Senior Engineers that are worth their weight in gold provide the role of mediator. I provide such a role in my little microcosm, and in turn answer to others when the problem steps outside my microcosm. In my organization, I have these groups all involved in changes to production: developers, system administrators, testing/QA, purchasing, security officers, storage administrators, network administrators, database administrators, and core infrastructure (electric, HVAC, floor loads). As a senior engineer,

    • I could not agree more. We have a fairly large projects where the devs do the deployments and I can tell you its all a big mess. Since the devs have the right to do deployments, naturally they can make small changes to the production environment invisible to the operation team. Since there are quite some incidents occuring on the production environment, the dev team tends to fix the problems on the production environment on the spot because "oh, its just a matter of fixing this and that", therefore the acce

      • Re: (Score:3, Informative)

        by cerberusss ( 660701 )

        Since there are quite some incidents occuring on the production environment,

        Well, it obviously doesn't matter enough for your customers to have those projects 24/7 online, otherwise things would be different. So why bother?

        In a previous job, we had three groups: development, system administration and application management & testing. Development would put the deployment files in a share and then sysadmin would take over, deploying it to acceptance and production if fiated by application management.

    • Yes, theoretically. However, that would also imply that IT makes all the specific requirements of their setup known to Dev.

      At least, it sounds like the deployment environment is rather tricky. If it's sufficiently tricky to have implications for development, then that should lead to explicit, testable requirements.

      • Tasks to deploy to the live environment should 100% match the tasks to deploy to the acceptance test (or staging, or whatever) environment.
          If you don't have a proper test environment, then the problem is in this fact, not in dividing Dev/IT responsibilities.

    • Re: (Score:2)

      by Rycross ( 836649 )
      Yep, thats how we do it at my place of work, at a very large bank. I wrote some scripts that create an installation package, which we developers run whenever we want to do a deploy. We then hand the packages off to our deployment team, who installs them on the server. We usually have a developer or two who are around to help debug configuration issues when they're working on a deploy. Also, I keep in pretty close contact with our deployment team to make sure that they are aware of changes in configurati

    • I work for a pretty big place [irs.gov] and while I agree with you, we've found that it's nice to have some IT folks who work closely with dev to usher new products to the field. We have a testing lab where dev hands off their work to a mixed IT/dev team (mostly IT) who then upgrade a sample network with the new product. They test against every expected configuration of hardware and software the field is likely to be using. If there are no problems, there's a joint sign-off, the testing team writes deployment inst

      • Re: (Score:2)

        by mwvdlee ( 775178 )
        So the deployment instructions aren't tested before deployment?
        • Deployment instructions are roughed out by the devs and handed to the testing lab. The lab does the trial deployment and if everything goes well, they finalize the instructions. Full deployment then follows. Normally, deployment instructions will continue to be revised based on field feedback throughout our deployments (which are often phased geographically, making it worthwhile and timely to continue upgrading the docs.)

    • Re: (Score:2)

      by THEbwana ( 42694 )
      Agreed.
      Also, in most companies I've worked which have to be SOX compliant dev's are not even allowed to connect to any UAT or PROD machines.
      For many industries (my experience is primarily within banking and finance), allowing developers access to prod or uat machines can actually be illegal and can many times be something that gets you fired...
      If the poster wants to be Sarbanes Oxley compliant - then allowing Developers to access PROD or UAT environments is a big nono.
  • ...or at least, should, but we're close enough, is to just provide releases at certain points and when we make a release provide one script to update from (at least) the previous version.

    That script (which probably calls other scripts in turn) should run backups, install new files, update the database schema, import any new data, the lot. Sure it's not trivial to produce such a script and they can end up taking on a life of their own but it saves the devs from having to talk to the IT monkeys.
    • If your IT department is staffed by monkeys, I suggest you get a non-zoo job and work at a REAL company. While I do know an unfortunately large number of incompetent IT people that shouldn't be working with computers or networks at all, none are monkeys.
  • When your team rolls out an application update that was created by an external company, Microsoft for example, do their coders pop in and lend a hand? No. The dev team should write an application effectively as a third party external to the rest of the company. If they're writing things that the IT dept can't roll out without their help then they're not doing a good enough job.

  • Production Services (Score:3, Informative)

    by bihoy ( 100694 ) on Wednesday December 13, 2006 @08:18AM (#17221196)
    This is why many companies start a Production Services team. Generally this means the hiring of a Build and Release Engineer or Manager who has an IT background and an understanding of Software Development.

    The alignment of the Production team varies. At some companies they report to the development organization (e.g. to the Manager or Director of Software Engineering) and at other companies they report to Quality Assurance.

    I would suggest that you check out the site: http://cmcrossroads.com/ [cmcrossroads.com]

    • Re: (Score:3, Insightful)

      by MoralHazard ( 447833 )
      You are 100% correct. I don't know what this guy's problem is--are IT and development are the only two departments allowed by law in his jusrisdiction? I mean, it's normal for an "Ask Slashdot" question to be totally stupid, but this one is pretty bad. I wish I knew what company it was, so I could avoid them. Someone always needs to lead a deployment project, and to be responsible for both the quality of the application AND the quality of the installation it's running on. This dude doesn't need a who

    • Re: (Score:2)

      by Rycross ( 836649 )
      Thats my main job description, and I'm mainly aligned with the development team (I report to the production support team). Good advice, especially when you work on large products that involve a lot of people.

    • Re: (Score:2)

      by Fastolfe ( 1470 )
      At a minimum, a company needs a semi-independent Production Support organization. The problem I have with ours is that nobody ever took the initiative to decide, at a high level, who would get what responsibilities. Consequently, Production Support tries to do the minimum amount of work they can while achieving the minimum amount of liability in the event something goes wrong. Ultimately all they do is set up and upgrade servers, and click the button to deploy software to production servers. The develop
  • This is why you should have at least 3 enviroments, dev, uat, prod (or similar) the dev's can carryout the promotion between dev and uat and in doing so iron out any bugs with their code / the promotion process. The entire promotion to production should be fully handed over to the production / IT team. At the end of the day you dont want developers anywhere near the production system.
    • In my workplace, the usual doctrine is that the transfer to UAT environment is also done only by the production team - in that way it ensures that 1) the installation instructions are clear and complete (or the tests would fail) and 2) you always transfer to live the exact same thing that was tested (instead of something with a 'tiny bugfix' added)
  • It depends on the the makeup of your organisation:

    Do you do ITIL?

    Do you have SOX obligations or something similar if outside the USA?

    Are you agile?

    Can you automate your releases?

    What technologies do you use?

    My organisation is a large user of IT but we are not an IT company (we're a University). We have a network of about 8000 Computers (Solaris, Linux and Windows) and are heavy users of Oracle products.

    We have a focus on workflow automation at all levels in the business and have around 200 IT
  • You work for a small company. Any advice presented by /. will fail to solicit the correct response---except this one. This fight is between department heads, and should be resolved between them as businessmen. Failing that, whoever controls both managers should make the decision. This is not a software issue, but a business decision outside your control.

  • Ommm... (Score:4, Insightful)

    by MarkusQ ( 450076 ) on Wednesday December 13, 2006 @08:33AM (#17221306) Journal

    In art classes they teach students to draw the space around the objects they are trying to depict. It's a useful skill in many areas.

    Rather than imagining that there is this atomic transition point that one side or the other must own, look more closely at what happens when changes are put into production, zooming in until you have enough detail that every piece naturally belongs to one team or the other.

    Then look at how this would play out in the real world, to find the "frothy" or "tangled" parts (well, IT should do this, then Dev should do that, then IT should do two more things, then it's Dev's turn again). These parts should be sorted out by requiring documentation (or scripting) to flow one way or the other, so that the process can be performed by one group without the direct involvement of the other.

    In short, the problem here is the granularity of your question.

    --MarkusQ

  • Most projects of a size to really worry about this stuff would have an Integration/Deployment team. Essentially they are the "middle ware" of the Application Dev cycle. Dev don't know about the realities of real implementation and have all sorts of bad habits which don't scale to production and IT/production don't know about the deeper technicalities otherwise they would leave and get jobs in Development!!
    Your Integration Team should be able to keep your dev/project "real", get them doing your Dev environ
  • ...you should make a decision and stick to it. It doesn't really matter which team is in charge, so long as one of them is and that the other team knows it.

    Bob
    • you should make a decision and stick to it. It doesn't really matter which team is in charge, so long as one of them is and that the other team knows it. ....said like a true PHB.

      • ....said like a true PHB.

        More like said like a developer who wishes managers would get on and do what they're paid for instead of letting dev and IT fight it out for themselves.

        Bob
  • I work in my city (population approx. 200,000) government's IT department as a developer. As someone else mentioned, the best way to handle code deployment is over a three-tiered system. At DIT, we have Dev, Stage, and Prod for the Internet, Intranet, and Applications sites. Developers should be given full read/write access to dev, in which they do all of their work. In no way should the world, nor the client, see dev. Upon completion, we promote our work to staging (via sending a promotion request, or doin

    • Re: (Score:2)

      by Fastolfe ( 1470 )
      If not more. We routinely have 4+ tiers. One for developers to experiment. One or more test tiers for developers to integrate all of their code in preparation for a release. (Sometimes multiple releases could be going on simultaneously.) Separate stage and production tiers (similarly if not identically configured) allow for pre-release testing and the release itself. The stage tier, if configured identically to production, also allows for stress/performance testing. If you have lots of similarly-conf
  • in our company. Once the developer(s) have a stable release, passes developer unit testing, it goes to QA (Quality Assurance) for systems testing and User Acceptance Testing. Once the user signs off QA asks dev for deployment procedures/scripts. Once Deployed the end user is asked for final sign off on the deployment. Software development is a TEAM effort requiring developers, DBAs, System Admins, QA, and end users. The only one of these that may not be IT is end users.
  • Each week we hold a meeting (call CCB - Change Control Board) where the people who do the development talk about what their projects that need to go in production in the upcoming week.
    All software & hardware changes will be discussed; what the original problem was, what the solution is to do, what changes to make, what effects to expect, what the rollback plan is, emgency support, and who's all signed off on the changes & tested them. It's then approved or denied (for lack of planning, support, use
  • in bigger companies, it's a CM team that handles these things. neither dev nor it can do the deployment. CM will testify that the release has been tested (which can be done by a testing group in dev, or a test department), and they will have a handle on everything that is installed right now and has ever been. They have the final word on whether something should be deployed. They build the packages going to the clients, if need be. And you can always go to them and ask "where's the code that's running right
  • Sooner said than done, of course... but if applications are so interdependent that the combined system is fragile unless everything is just exactly the right version and installed just so and configured just so and located in just exactly the right place in the directory and has all the other configurable settings of everything else in the system just so... then it wasn't well designed.

    In the Good Old Days an application was a single file, you copied it to your system, and ran it. Over the years--I tend to
  • ...The IT person(s) who will be hadling deployment should be involved at all stages of Dev. They should attend all development meetings, and be part of all discussions that will affect deployment.
  • Why would anyone want control of a balky, high profile and critical process? Glory? Future generations of IT managers speaking your name in tones of hushed awe?

    Kidding aside, I've worked both ends of that stick. If anybody is goign to "own" a deployment it's going to be IT. But owning doesn't mean you don't share. IT and development both have roles in the deployment that, if you must insist, belong to one or the other. IT should make development aware of any deployment constraints, the development peop
  • we have someone in our company that is a middle ground between the dba's/IT and the dev's. He is a coder by training but he supports the code the developers write, deploys it, checks for problems after QA signs off on it, etc.
  • the same people who write the code where I work also take care that it deploys properly. Sure someone else pushes the buttons and writes some user information but until its in the field at all locations the developer is actively involved.

    code goes from development, q&a, field test, deployment.

    once the code reaches q&a it does not change unless q&a rejects, but afterward the developer(s) are responsible if problems arise during install and work with support to fix it. they never lose ownership.
  • I work for a very large software company and we're going through this battle right now for an internal application.

    The developers are fine at making improvements to the application interface and providing new features, but completely and utterly clueless when it comes to system architecture, security, using hardcoded hostnames, etc. etc. The more we close the developers out of the production deployment, the better this runs. We also have to increasingly review their work to make sure they aren't doing thi

    • Re: (Score:2)

      by j-pimp ( 177072 )

      The developers are fine at making improvements to the application interface and providing new features, but completely and utterly clueless when it comes to system architecture, security, using hardcoded hostnames, etc. etc. The more we close the developers out of the production deployment, the better this runs. We also have to increasingly review their work to make sure they aren't doing things like hardcoding hostnames, or writing data to local disk instead of the rdbms (writing data to the disk kills horizontal scalability, high availability, etc)

      It sounds like your developers suck. Did you try educating them? Suggesting better ptactices?

      • Sounds to me like the managers suck. They should be seeing these things happening, and creating better coding guidelines and working with the developers to improve their practices. No guidance from management means no improvement ever.

        • Re: (Score:2)

          by j-pimp ( 177072 )

          Sounds to me like the managers suck. They should be seeing these things happening, and creating better coding guidelines and working with the developers to improve their practices. No guidance from management means no improvement ever.

          Well its a two way street. The developers and IT personnel should point out flaws in the process to management and suggestions for improving them. Management should also be looking for ways to improve the process on their own.

          Find a quick win proposal that can produce results management and all other parties can see. Get them to implement that successfully and they will listen to your crasy ideas of making an installer program. BTW, once you get your installer working, it usually needs very little mai

  • Speaking as a Release Manager... (Score:4, Insightful)

    by Horza66 ( 1039328 ) on Wednesday December 13, 2006 @10:12AM (#17222252)
    Plenty of other posters have pointed out that you sound a like an operation that is a bit small for the full Software Development Process. However if you're asking I suspect you're a growing company, in which case you need to get a Process in place, and soon or you will experience the full agony of a chaotic IT environment. (NB That's where I work now - I've worked sane places too) Fairly typical Process: 1. Dev receive Requirements and Defects from the Business, and code to them, unit testing their code (). 2. Code is delivered to Operations with a 'Release Note' or equivalent covering how to deploy the code to Environments 3. Operations deliver (deploy) the code to test environment(s). Link and Acceptance testing is performed - does it meet Requirements? are key defects resolved? Plus regression testing - does it break the existing system? Test sign it off if it clears these tests. 4. Operations deploy the code to the Production system on sign off. You inevitably end up with tensions in the Business vs IT, plus the divisions between the priorities of Dev, Test and Operations. Sounds like you are at the stage of not having any well-defined roles/teams for these responsibilities. I'll detail the Operations breakdown too. Operations: As others pointed out this breaks down into various teams. DBAs, Sysadmins, Change Management, Release Management, Operators, depending on your site. Operations are responsible for the stability and smooth running of the Production system - they must accept change, but control it. Since I work there, and you specifically address the subject, I'll detail Release Management too Release/Change Management Usually end up the Gatekeepers on changes. They'll need to be familiar with the whole system, and resistant to the pressure they'll receive from all sides. They need to know what versions of code are where, and be able to reject bad code when it turns up, but be flexible enough to make sure Test have something to test. They need to be experts on everything your IT does. No jobsworths here, and good generalists are rare. Since you'll inevitably go through a period of chaos if you are growing I'll mention that staff turnover here is very high - unless you get in contractors, and pay highly for them. The Change Management role, sometimes covering the Release role too, is to track changes and know who they impact, and be able to prioritise changes. If Release and Change roles are separate, CM is closest to the business. Hope that helps.

  • I think it depends. Where I work we do multiple deployments: into the common development environment, into the QA system, into an internal user-test system, into a customer-test system and finally into production. The rule is that devs do the deployment into the CDE to work out the kinks. Devs and QA do the deployment into the QA so QA knows what's going on. Devs work with the IT staff doing the deployments into the two test systems so that IT knows how it goes and devs can work out any final issues that co

  • Root. Root owns it. Root keeps everything that doesn't come from us running. Root, Chief Senior Root, Senior Root and 2nd Root keep a Binder and a Booklett with all procedures and Accounts written down, in case they all die in a planecrash. Dev builds the stuff deployable and is assitant to Root on deployment and irons out the glitches as Root orders. During Developement it's the other way around. Dev tells Root what it needs, Root delivers. If the System goes down, Root takes blame. Dev owns Root, but Root
  • The CIO or whomever makes the IT policies should be the one to say who "owns" the deployments. That is what management is for, making these kinds of decisions. In a way, the person who should "own" the deployment is the Project Manager.

    This decision should be enforced by the Project Managers (PMs) overseeing the projects. If your company doesn't have PMs overseeing the projects, then it may want to start assigning them.

    The PMs are the ones who are supposed to be keeping the changes on track, on time, and on

  • Unless your IT department is under-educated, you should have a number of IT guys who are moderately capable software developers themselves, and at least some of your developers should have some system administration skills as well.

    Given that, one good solution is obvious: For each development project, put someone from IT on the development team. The development organization is free to assign that person some development tasks, but their *primary* job is to ensure that development is taking IT's concerns

  • It should be IT deploying to production period (I'm the Sr. System Admin and deploy all apps to production). Yes we know our own systems and we know the application to an extent. If done right "code monkeys" have done their job and the application passed QA'ed and testing, the application then belongs to IT. In my experience the vast majority of DEV's don't understand System Administration (there are exceptions to the rule of course) and no it's not about us loosing control but boils down to accountability.
  • The answer is: "neither". This is why you have a project manager (I hope.) Someone neutral who tracks progress and assigns tasks to dev/IT/whatever.

    At least I hope you have a project manager. If not, assign one who's capable of working outside his normal role, either dev or IT, but they'd better know what they're doing (cost tracking, that sort of thing.)

  • Here at our shop, there is a gradual transition of ownership. I'll replace IT with Operations for us, because we're actually a tech company, and our IT department is the internal computers-on-desks group, where as the Operations team is the group that manages all fo the customer-facing money-making production sites.

    Development owns their development environments. We provide as-can-do support for them, during business hours, when nothing else is going on. From there, a build manager (development does not

  • The arrangement I tend to advocate revolves around four teams:

    Development
    The guys producing software. They shouldn't need to know anything about the servers, though potential server configurations need to be part of their requirements, so they don't do something stupid that prevents them from running. Software defects go to this team to be resolved. Things like data sources, remote services, usernames and passwords, etc., are specified through configuration, not hard-coded.
    Deployment
    The deploy

  • Development gated by QA (Score:3, Insightful)

    by GryMor ( 88799 ) on Wednesday December 13, 2006 @01:03PM (#17224688)
    IT is strictly responsible for low level infrastructure (OS, hardware, physical network, power). Development teams own services and are responsible for their fleets in both a development and operational sense, and is responsible for notifying their upstream and downstream dependancies of changes in advance. Actual deployment (which, if it requires documentation, is not being supported by a sufficiently advanced deployment management system) to production is gated by Development's QA teams, who are responsible for testing on non production systems.

    We used to have dedicated deployment engineers, but that just added friction, and guarenteed that the person doing the push to prod didn't know the full contents of what they were pushing.
  • At my company we have a clear de-markation Dev does dev and IT looks after the env, including OS and all installs.

    IT has it easy beacuse we have an automated install process which includes roll backs etc. The only time we have any sort of issue is when the automated install software needs to be re-configured. Then two people one from IT and one from Dev get given the task of developing and implementing the change.

    Basically, promotion to production is a non-isuue due to automation.

    Now if I could just get t

  • Neither owns deployment (Score:3, Insightful)

    by Aging_Newbie ( 16932 ) * on Wednesday December 13, 2006 @01:48PM (#17225292)
    So, how should one deploy changes?

    1. Dev completes their changes and makes a release including operational details as needed.
    2. QA/Testing roll the package to their staging environment and complete their testing. Pass goto 3 fail goto 1
    3. Configuration Management (usually part of QA) releases the package with installation instructions
    4. IT follows the instructions and rolls the application to the live environment
    5. QA tests the operation in live and reports the status for a go/no-go on the changes

    DBAs should package their changes in the form of repeatable scripts that are used to move the code and data to Staging, and Live. That reduces variability. Most DBAs already know the impact of their actions so they can perform the moves as requested by QA.

    Now, before you ready the tar and feathers, it is possible to plan orderly releases that follow that process and it produces near zero failures in production. QA's job is to be the interface between the development activity and the real world. They have the discipline and skills to follow processes and keep bad things from happening. But QA has to have the power to call the shots.

    If you do this ...

    * Developers win because they no longer hold the bag for consequences of bad changes.
    * IT wins because they know precisely what is going on and they are empowered to fix or restore stuff because they know exactly how to install the code without breaking something.
    * Project managers who carefully orchestrate the whole process earn their keep.
    * Micromanagers and others who like to call for quick hit changes to cover up for disorder and disarray somehow find their habits have no place in the organization.

    Customers will be much happier and willing to accept slower and more orderly propagation of changes when they realize that they get better quality and uptime. Most of the pressure on development comes from emergency recovery from avoidable errors rather than actual work to be completed. One could argue that if the time from a request to acceptable code is measured, the prcess saves time overall.

  • I've often seen people try to split it up this way: dev creates the software, sysadmins (or IT or operations or whatever your company calls it) owns it after deployment. It seems logical to have people who are responsible for overseeing the network and machines, and having them install and run all the software on those machines sounds reasonable too.

    In practice though, this has always led to a very adversarial relationship. The sysadmins probably don't know much about how the software works, so they end

  • In our company, everyone is on the same team. That is to say, we avoid the split between developer and IT staff, and instead consider us all as part of the team working on a particular software solution.

    Just fixing this psychological schism helps quite a bit, as now we all work together to solve all those problems that always arise when you face a new deployment.

    Of course, it also helps that we are a small enough company that you don't have to compete with anyone else in order to brown nose with management
  • If you're asking the question, you have already gone wrong. Seriously, if the organization divides the functions and has fiefdom wars going on like this, you're sunk and someone higher than you will have to fix things.

    In places where I've worked where this divide exists, I just don't get into the dispute. I do my work and let the office politicians worry about the next step, unless I'm specifically asked for my input.

    By 'do my work', I mean do it right. In one environment where IT did the deployments, I got
  • You need to get your DEV to generate a package of software that's installable against their last release. That package goes to your QA to install in the QA environment as per the installation instructions that go with the package. If QA says the package is good send it up to your production/application/sys admins to install on the live system.

    The key is to keep the process simple but never, ever let your DEV have access to your live production system unless it's a break/fix scenario and the admin is lo
  • The first rule of operations is, Development does not have any access to the production servers.
    The second rule of operations is, you can always roll back to the previous version without losing data.
    The third rule of operations is, you always, always write an installer.

    After that, the rules depend on how much money you have. In an ideal world, you've got 4 environments:

    1) Dev. Minimal number of servers to build and test the app. Ops has no logins here, dev runs the show, they take care of sysadmin here

Slashdot Top Deals

In order to dial out, it is necessary to broaden one's dimension.

Close