Ask Slashdot: System Administrator Vs Change Advisory Board 294
thundergeek (808819) writes "I am the sole sysadmin for nearly 50 servers (win/linux) across several contracts. Now a Change Advisory Board (CAB) is wanting to manage every patch that will be installed on the OS and approve/disapprove for testing on the development network. Once tested and verified, all changes will then need to be approved for production. Windows servers aren't always the best for informing admin exactly what is being 'patched' on the OS, and the frequency of updates will make my efficiency take a nose dive. Now I'll have to track each KB, RHSA, directives and any other 3rd party updates, submit a lengthy report outlining each patch being applied, and then sit back and wait for approval. What should I use/do to track what I will be installing? Is there already a product out there that will make my life a little less stressful on the admin side? Does anyone else have to go toe-to-toe with a CAB? How do you handle your patch approval process?"
I do this (Score:5, Interesting)
I have to do this and it's no problem at all, although our change management process doesn't sound quite as onerous as yours (I suspect yours will adapt over time -- the CAB will soon get bored if they have to approve every single OS patch).
I have to do a risk analysis for each change that gets made to a system (not just patches). Sometimes this risk analysis is fairly informal, for example if the change is to add more RAM to a VM, it's very unlikely to have a significant adverse impact and is easily reversible, so low risk. Other times the risk analysis (and processes that come out of that) may take a long time and require significant co-ordination with other parts of the organisation I work in.
A good example is if we make a change to a service that impacts the look and feel of that service. It will require co-ordinating with our communications, helpdesk, training and documentation teams as well as other parts of the technical group I work in and the CAB really acts as a check to make sure all of that has happened properly.
There are still a few people in our organisation who see the CAB as a barrier to getting work done, but for me it is really a check to make sure we're delivering changes in a proper way.
I can recommend you take a look at The Phoenix Project by Gene Kim, Kevin Behr and George Spafford. http://itrevolution.com/books/... [itrevolution.com] - I had quite a few "this is where I work" moments whilst reading it :)
Re:Nonsense (Score:5, Interesting)
Re:Nonsense (Score:4, Interesting)
Can't find it on Google or Bing though for some reason.
Re:They have a point (Score:4, Interesting)
As a software developer I have multiple times had a development box screwed over by an IT department pushing unneeded drivers and patches that cause problems.
I say prove they are good or needed before you waste other peoples time.
If you just want to push any random patch that comes along then you should be forced to resolve all issues without the traditional reinstall the machine.
Er, waste people's time?
As a software developer, you have no fucking idea how difficult it is to pick and choose patches and driver updates to get pushed out to machines while also trying to maintain any sort of consistency with patch levels across the enterprise, but apparently this is something you want me to waste my time on in order to ensure you've not lost a spare second on the rare and random occurrence that you experience a problem in 1 out of 200 patches (my patching track record over 15+ years shows the frequency quite less than that)
And if you're doing patching correctly, you're mainly concerned about patches deemed "critical", so again, you're not really afforded the luxury of picking and choosing here without risk.
As a seasoned sysadmin, I have a fix for you. It's called VMs. Play till your hearts content and press the rewind (snapshot) button when find the environment screwed up (shockingly, it's not always the IT department that screws computers up...yes, I know this is breaking news)
I'm not a fan of CAB (Score:3, Interesting)
I used to work for a Fortune 100 company. I'm not sure how CAB works at other companies but I get the impression that their implementation was flawed. 1) You could easily go around the process. 2) I'm certain nobody reviews the code - They just kind of discussed it. In my opinion this is a half-baked solution to prevent things from getting pushed to production which could cause problems (errors, leak sensitive info, etc). I am 100% confident that I could have gotten CAB approval for nearly anything. I understand the idea behind CAB but in my experience it isn't effective.
I actually quit that job partially due to things like CAB. Increasingly control was taken away from people in the IT department, and handed to things like CAB or to 3rd parties who managed our systems, databases, etc. The jobs of myself and others in IT staff were being reduced from "actually doing the work" to "submitting tickets and following up on tickets." Nothing like being on hold when calling the 3rd party for a critical issue you yourself know how to fix in 5 minutes. It's also a blast when I had to tell the support guy what commands to run because he wasn't familiar.
And no we didn't fuck up anything to deserve this treatment. It was dictated to us from upper management.
Re:As a Change Manager... (Score:4, Interesting)
OP is managing 50 servers, you are managing tens of thousands of Systems - the situations are hardly comparable.
Like the OP I am the sole admin for our companies IT (60+ on prem servers, mix of WinTel and Linux, plus 10 Azure hosted servers) and I am in charge of patch management.
If a committee in the organisation came and told me they were taking responsibility for patching away from me I would either tell them to sod off OR I would hand over all the admin accounts and wish them luck.
Pre-approvals (Score:4, Interesting)
Big company experience comes to small company (Score:4, Interesting)
Yes, I know how they are thinking and the pain you are feeling. To accomplish the implementation of this change management process you will need a lot of people working for you. Use this to your advantage. Quickly study up on the subject so your experience with the systems will not leave you with a dog pile of new bosses to tell you how to do your job. Instead insist that you need to hire more people to manage the overhead.
In the end that probably won't work and you'll be kept "at the bottom" where you are now.
These changes are going to be enormously expensive and despite all you have done, it will be perceived that you created this mess by not having a change management system in place to begin with. Of course, they will also see that you don't know about change management and will prefer to hire someone who already knows about it.
Now I'm not going to down change management processes. They can prevent problems and identify people who would otherwise deflect blame and hide in the shadows. But from what I have seen, you're just getting the beginning of the tsunami of changes.
Push for testing systems and additional hardware to support it. Of course it will also require more space and other resources. Try to get ahead of this beast.
Re:Nonsense (Score:2, Interesting)
This. We have a CAB. We got patches to be considered a routine process with specific scheduling - DEV on day X of the month, QA on day Y of the month, and Production split between days A and B. Submissions to CAB for automatic patches are FYI and made in batch.