Slashdot Log In
Who Owns Deployments - Dev or IT?
Posted by
Cliff
on Wed Dec 13, 2006 06:45 AM
from the custody-battles-are-always-nasty dept.
from the custody-battles-are-always-nasty dept.
txpenguin asks: "I am IT manager for a small software company. We host several generations of our applications in a fairly complex environment. Our systems are very much inter-dependent (clustering, replication, heavily loaded, and so forth), and bad changes tend to have a domino effect. Additionally, it seems that there are always those who need to be 'in the loop', but aren't aware of changes which affect them. There is a constant battle between IT and Development regarding who should handle the deployment of new code releases and database schema changes to production systems. Dev doesn't understand the systems, and IT does not know the code well. How do you handle this at your company? What protocols seem to work best? Can there be a middle ground?"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Middle ground (Score:5, Insightful)
You have to compromise. That's it. Middle ground. There are no other solutions to or ways around this problem. As you describe it, each side has access to and knowledge of half the problem. Half plus half is whole!
So, meet with the guys in Dev. If you want to be beaureaucratic and official about it, create a "deployment team" consisting of an equal number of members from each side that will sit down, discuss, and supervise all necessary changes to production systems. Hell, send someone to a project management class if you need to.
Now, the obstacle you're likely to hit is office politics. People won't want to listen to others and/or won't want to give up their turf or allow others on it. Too bad. To place how serious this issue is in overcoming the political terms: everyone in both departments needs to be cooperating or unemployed.
So there you go. Just like any other relationship, business or otherwise: sit down and talk it over. Problems solved!
Re:Middle ground (Score:4, Insightful)
Developers write code & documentation.
Installation/Deployment guides *are* required documentation.
IT takes the software, and follows the guide.
Business applications are mostly consolidated on a few servers. IT guys know dependencies, time windows, batch runs and the whole shebang. A dev team has no business doing all this.
Parent
Re:Middle ground (Score:5, Funny)
Developers write code & documentation.
Installation/Deployment guides *are* required documentation.
IT takes the software, and follows the guide.
And in Magic Happy Land, that actually works without a problem.
Parent
Re: (Score:2)
But it doesnt have to. That's why in a good shop, there's actually a process for rolling-back the install, contingency plans, business continuity and disaster recovery. It's not a trivial process; even more important that there's a clear separation of responsibilities.
Re: (Score:3, Insightful)
"rolling back the install" is a low-grade disaster recovery scenario. Testing the install on a non-production machine, working out the install/upgrade kinks and maybe even having a team of testers or some scripted testcases to throw at it before you start doing anything on the production systems is disaster prevention.
And any doctor, sysadmin, or person with a modicum of common sense (or at least familiarity with some common-sense aphorisms) will tell you something about the
Re: (Score:3, Insightful)
And in Magic Happy Land, that actually works without a problem.
That land is called Sarbanes-Oxley Land and it has to work or you fail your audit.
Re: (Score:3, Funny)
Re:Middle ground (Score:4, Informative)
--Mike
Parent
Re: (Score:3, Insightful)
Installation/Deployment guides *are* required documentation.
IT takes the software, and follows the guide.
And note that these are guides, not step-by-step instructions. They should say things like "Load the database schema update script (app_schema_updt_1_1_19.sql) into the database". The actual mechanics of doing this (making a backup, bringing an additional transaction log file on-line, starting table-level auditing, whatever) are left to the people who actually know the system (and the department procedures). Part of the development process is circulating a draft of the installation guide to IT for comme
Re:Middle ground (Score:4, Insightful)
So. you have a certification team (or quality team, or test team) who's job it is to certify that what dev has goven them works as dev said. These guys install it on their own separate systems that mirror the business (on a smaller scale) and test it out. Bugs get reported back to dev who get to fix them and so on. Eventually it'll get rolled out to IT who will have a reasonably good expectation that it'll all work.
However - even in the best of cases there will be exceptional circumstances, and its at this point that IT will get dev members to come and fix up issue that arise on the live system. IT should be first contacting the cert team who will determine the bug (hopefully with a bit more inside knowledge to reproduce it on their systems) and will then get dev to issue a patch, which goes through the standard release process.
Of course, if you want to let dev team hack about (which is probably why you have such a complex system in the first place), and IT to twiddle with their setup then fine - expect it all to go arse-up.
I like to think of these environments as always having a 'customer' that they deliver to. If they provide a poor service, the customer has every right to complain. So, Dev's customer is the IT guys, IT's customer is the Business, and Business answers to real, paying customers. Such a chain of responsibility does focus people's attention on what they are trying to achieve for the company.
Parent
Re: (Score:3, Funny)
What about QA? (Score:4, Insightful)
When we have a RC I'll branch the trunk, and request that QA perform a Pre-Production build. Developers will work with ops to get this running properly on the pre hardware, as this can be done outside of maint hours. We'll then do several builds of the branch until it's gold, and then tag off the branch as X.vv.zz.
While a major release is in QA the lead focuses on creating/updating the operations document which addresses day-to-day maintainence issues and tells operations how to troubleshoot the app in the case of a problem. They also produce an implementation plan which identifies the groups/persons needed to deploy the application, and the steps needed to be taken, using what they've learned from the initial pre deployment. Once this is done, and QA has promoted the app, a dry-run is performed to try to catch any missing steps. The implementation plan is handed to QA, who coordinates with IT/Ops to resolve any conflicts and schedule the deployment. Ops/DBA's then physically performs the deployment following the steps given in the plan. In a major release situation, you may have a team lead or platform manager coordinating the actual steps on a conference bridge. But for minor releases we've been able to just have our operations teams do the full deployment with verification by QA and the product's customer service group.
We also have a twice weekly meeting where any upcoming production changes are discussed between IT/Ops, QA and Dev. Release documents are put on a calendar, so if an issue comes up on another product we can go to this and see what may have caused it.
Dev and QA also meet weekly to discuss the progression of products through or into QA. Any issues with testing or problems with builds not being stable can be addressed.
It took us a while to get to this point. We had previously been in a situation where dev would handle the build and deployment process, and it was had for many of the leads to let their projects go, but now we can see the benefit, not only for the company, but also in the fact we don't have to be doing releases at 12AM on tuesdays anymore. It takes a lot of work across departments, and definitely is a long road, but something that needs to be done.
Parent
Speaking as a developer (Score:5, Insightful)
Assuming the dev department does their job well, a deployment would not require any of the dev department's skills.
Re: (Score:3, Informative)
Re: (Score:3, Insightful)
Re: (Score:2)
Short of all that, they're going to need to work together on something.
Re: (Score:2)
This is why we have something called a QA department
Re:Speaking as a developer (Score:5, Interesting)
Absolutely agree.
If the developers do their job correctly, then a release should include a full set of install and migration instructions for IT to use.
If IT do their job correctly, they should test the install on a separate system before deploying it live.
If the install does not work 100% first time, don't tweak it, send it back.
If the developers complain that IT didn't follow the instructions correctly, then the instructions were wrong.
Send it back to the developers to write better instructions.
If all goes pear shaped on the live system, IT should have a full set of (tested) instructions on how to rebuild the system from scratch.
If the developers can't supply those instructions, then you don't know what you have.
Ok, I know this is nice in theory and difficult to acheive in practice, but both teams should be working towards this as their goal.
Parent
Mod Parent Up (Score:2, Interesting)
In a previous shop, we treated our in-house developer code releases like any other vendor release. Just because some code was written by the guy down the hall, shouldn't mean that I can't ask for the same level of documentation/support I get from another Tier 1 vendor.
Re:Speaking as a developer (Score:4, Insightful)
That's not always true, Mr. Nick Burns [wikipedia.org]. Sometimes IT has a permanent bug up their network port and refuses to learn a small amount of Developer's vernacular to share in the process. Likewise, Developers should not have to know how to speak 100% "IT" to write instructions. There is a common ground. IT personnel are paid for their experience, and ability to adapt, and not to simply follow instructions.
Parent
Re: (Score:3, Insightful)
Re: (Score:3, Funny)
I keep hearing people talk about "non-production hardware" - and it sounds like gibberish.
I asked my manager about this, and he snapped the pencil he was holding and started muttering something about "damn commies.". .
Yes - there is a "Fantasy World" - it's where people are promoted into management for competence, and allocate sufficient resources for projects, and blue fairies ride on rainbows throwing bags of candy.
In the real world, we "test" - hell, we DE
Speaking as an IT coordinator (Score:2, Interesting)
Re: (Score:3, Informative)
Well, it obviously doesn't matter enough for your customers to have those projects 24/7 online, otherwise things would be different. So why bother?
In a previous job, we had three groups: development, system administration and application management & testing. Development would put the deployment files in a share and then sysadmin would take over, deploying it to acceptance and production if fiated by application management.
Re: (Score:2)
Yes, theoretically. However, that would also imply that IT makes all the specific requirements of their setup known to Dev.
At least, it sounds like the deployment environment is rather tricky. If it's sufficiently tricky to have implications for development, then that should lead to explicit, testable requirements.
Re: (Score:2)
If you don't have a proper test environment, then the problem is in this fact, not in dividing Dev/IT responsibilities.
Re: (Score:2)
The way we do it... (Score:2)
That script (which probably calls other scripts in turn) should run backups, install new files, update the database schema, import any new data, the lot. Sure it's not trivial to produce such a script and they can end up taking on a life of their own but it saves the devs from having to talk to the IT monkeys.
Who does it with other apps? (Score:2)
Production Services (Score:3, Informative)
The alignment of the Production team varies. At some companies they report to the development organization (e.g. to the Manager or Director of Software Engineering) and at other companies they report to Quality Assurance.
I would suggest that you check out the site: http://cmcrossroads.com/ [cmcrossroads.com]
Re: (Score:3, Insightful)
Re: (Score:2)
Don't Ask Us . . . (Score:2)
Ommm... (Score:4, Insightful)
In art classes they teach students to draw the space around the objects they are trying to depict. It's a useful skill in many areas.
Rather than imagining that there is this atomic transition point that one side or the other must own, look more closely at what happens when changes are put into production, zooming in until you have enough detail that every piece naturally belongs to one team or the other.
Then look at how this would play out in the real world, to find the "frothy" or "tangled" parts (well, IT should do this, then Dev should do that, then IT should do two more things, then it's Dev's turn again). These parts should be sorted out by requiring documentation (or scripting) to flow one way or the other, so that the process can be performed by one group without the direct involvement of the other.
In short, the problem here is the granularity of your question.
--MarkusQ
As An IT Manager... (Score:2)
Bob
From a Dev Guy (Score:2, Informative)
Dev is IT (Score:2)
I work for a large HMO, and here's how we do it. (Score:2)
All software & hardware changes will be discussed; what the original problem was, what the solution is to do, what changes to make, what effects to expect, what the rollback plan is, emgency support, and who's all signed off on the changes & tested them. It's then approved or denied (for lack of planning, support, use
Stuff shouldn't interact (Score:2)
In the Good Old Days an application was a single file, you copied it to your system, and ran it. Over the years--I tend to
IT...However... (Score:2)
Perhaps I'm missing something (Score:2)
Kidding aside, I've worked both ends of that stick. If anybody is goign to "own" a deployment it's going to be IT. But owning doesn't mean you don't share. IT and development both have roles in the deployment that, if you must insist, belong to one or the other. IT should make development aware of any deployment constraints, the development peop
make a middle ground job (Score:2)
Why separate dev from it? (Score:2)
code goes from development, q&a, field test, deployment.
once the code reaches q&a it does not change unless q&a rejects, but afterward the developer(s) are responsible if problems arise during install and work with support to fix it. they never lose ownership.
IT for the win (Score:2)
The developers are fine at making improvements to the application interface and providing new features, but completely and utterly clueless when it comes to system architecture, security, using hardcoded hostnames, etc. etc. The more we close the developers out of the production deployment, the better this runs. We also have to increasingly review their work to make sure they aren't doing thi
Speaking as a Release Manager... (Score:4, Insightful)
Development gated by QA (Score:3, Insightful)
We used to have dedicated deployment engineers, but that just added friction, and guarenteed that the person doing the push to prod didn't know the full contents of what they were pushing.
Neither owns deployment (Score:3, Insightful)
1. Dev completes their changes and makes a release including operational details as needed.
2. QA/Testing roll the package to their staging environment and complete their testing. Pass goto 3 fail goto 1
3. Configuration Management (usually part of QA) releases the package with installation instructions
4. IT follows the instructions and rolls the application to the live environment
5. QA tests the operation in live and reports the status for a go/no-go on the changes
DBAs should package their changes in the form of repeatable scripts that are used to move the code and data to Staging, and Live. That reduces variability. Most DBAs already know the impact of their actions so they can perform the moves as requested by QA.
Now, before you ready the tar and feathers, it is possible to plan orderly releases that follow that process and it produces near zero failures in production. QA's job is to be the interface between the development activity and the real world. They have the discipline and skills to follow processes and keep bad things from happening. But QA has to have the power to call the shots.
If you do this
* Developers win because they no longer hold the bag for consequences of bad changes.
* IT wins because they know precisely what is going on and they are empowered to fix or restore stuff because they know exactly how to install the code without breaking something.
* Project managers who carefully orchestrate the whole process earn their keep.
* Micromanagers and others who like to call for quick hit changes to cover up for disorder and disarray somehow find their habits have no place in the organization.
Customers will be much happier and willing to accept slower and more orderly propagation of changes when they realize that they get better quality and uptime. Most of the pressure on development comes from emergency recovery from avoidable errors rather than actual work to be completed. One could argue that if the time from a request to acceptable code is measured, the prcess saves time overall.
Re: (Score:3, Informative)
So what about someone like me in the banking industry? You have any idea how long and hard the auditors will bitch if there isn't a separation of duties? Technically the developers are not to have ANY access to the production environment above that of an end user. (operative word being technically)
Re: (Score:2)
An end user is an officer of the bank that is trained and authorised to transfer real money to/from customer accounts. No IT-developer has the right to even view the customer account balances. Some from IT-production technically can look, but Internal security guys do try to log and audit the records that they look at, and viewing a politician's/musicstar's bank account is con
Re: (Score:2)