Ask Slashdot: System Administrator Vs Change Advisory Board 294
thundergeek (808819) writes "I am the sole sysadmin for nearly 50 servers (win/linux) across several contracts. Now a Change Advisory Board (CAB) is wanting to manage every patch that will be installed on the OS and approve/disapprove for testing on the development network. Once tested and verified, all changes will then need to be approved for production. Windows servers aren't always the best for informing admin exactly what is being 'patched' on the OS, and the frequency of updates will make my efficiency take a nose dive. Now I'll have to track each KB, RHSA, directives and any other 3rd party updates, submit a lengthy report outlining each patch being applied, and then sit back and wait for approval. What should I use/do to track what I will be installing? Is there already a product out there that will make my life a little less stressful on the admin side? Does anyone else have to go toe-to-toe with a CAB? How do you handle your patch approval process?"
Patching.... (Score:5, Informative)
What we normally do is get a blanket approval if its coming from the OS provider with an understanding that patching will be done on a specific schedule.
IE. If all the patches come from Redhat there is no approval its necessary to keep them up to date for security purposes. The same is true for patches pushed out from Microsoft.
Then your only dealing with 3rd party applications. Even those the more common ones we get added to the blanket approval, ie. Adobe. This way you are only telling them you are bringing them into line with the latest set of patches provided by the OS vendor without having to list all the packages that are being updated. Then they only have to ask you if a program has or does not have a certain bug.
Setup a WSUS server (Score:5, Informative)
There are commercial products that can also this in a nicer manner but they cost money.
Are you still partying like its 1999, or what? (Score:0, Informative)
Well, welcome to the big leagues.
Any company of any reasonable size NOT doing something like this is stupid.
Not that I have any love for CABs, Change Management, quite the opposite.
However, when the shit hits the fan, someone is going to be doing an Root Cause Analysis, and having all that stuff available is useful/necessary/legally required in some cases.
You're not the only one out there that has to deal with this. Some places you need CAB approval via a Change Request in Remedy just to change port speeds.....
Some sort of Blanket Approval as mentioned earlier will solve a lot of the hassle, and let you minimize required Changes to a smaller subset of actions.
1 of 3 possibilities... (Score:1, Informative)
1 of 3 possibilities:
1. You are perfect. You NEVER screw up. In this case, the CAB is just being a PITA.
2. You can make certain types of updates quickly, with little or no risk, and you never screw up. The CAB should agree to make these standard changes with very low overhead. The other types of updates are likly to help YOU, not to mention everyone else in the company that depends on you.
3. It's hard to say in advance - most of the time, things work OK, but sometimes problems arise and there is unexpected downtime (it's NEVER your fault, however). Bit the bullet. You are not running a world class shop and you need help to improve. Anyway, downtime in production always takes more of your time than filing an RFC.
Re:Nonsense (Score:5, Informative)
They want bureaucracy, they make the paperwork. Tell them to track windows and distro security pages, the changes are there. I would be toasted with that kind of tape, I updated my servers in a pinch immediately after the first news of heartbleed at 3 in the morning. 0300AM right. How about dusting your resume and changing jobs? Let them play the shuffling reports game alone.
I've served on a change control board. Every application and system update was supposed to be bundled to make the sysadmin's job easier, include a document that outlined the nature of the change and why it was needed, the instructions on how to apply the change, and the instructions on how to recover if it didn't work.
Change committee met once a week, approved/scheduled, deferred, or rejected changes. In case of emergency, the CIO or designated proxy could approve an out-of-band change request.
We didn't attempt to micro-manage changes, just understand the business risks and rewards. Obviously, the more details you could capture the better prepared you were to understand the consequences and the ways you could recover. But when Microsoft hands you a CAB that includes patches for SSL, IE, 6 GDI bugs and Windows notepad, that's their problem, not yours.
The one thing that we didn't do (obviously!) was allow automated Windws updates. Then again, considering the damaged that some Windows Updates have done to desktop machines, I didn't even allow that on my desktop machine.
What went wrong before? (Score:4, Informative)
In my experience a CAB usually gets introduced in a small organization if something really got screwed up under the old process. There are exceptions - you could get a CTO who is gung-ho for ITIL, or you may have a new, important customer who insists on "process". But a CAB is an attempt to manage change and prevent problems in the working environment. So unless you have a better solution that will prevent negative impacts from your change process, go do the paperwork, with special attention to any risks or issues associated with the change (extended maintenance window, complex install or backout process, partial or incomplete fixes that still leave issues open). You can probably half-ass the CAB and get your work done almost like the old days, but when the next failed change occurs and they find out you hid risks or didn't do proper research, your ass could be out the door.
OTOH, if you really hate bureaucracy that much, hauling your ass out the door could be your best option - as long as you have a different career in mind besides sysadmin.
Re:As a Change Manager... (Score:4, Informative)
Exactly. While I wouldn't have a problem if it was just a matter of tracking when changes were applied for audit purposes, having to document each patch is completely unreasonable. In my case, we have various regulations that require us to patch our systems within a certain period of time, which we've translated into patching once a month. If I get asked for justification for installing a particular patch, it's usually "because if I don't the auditors will ding us for not installing it." If something needs to be done off-cycle or in an emergency (e.g., OpenSSL updates for Heartbleed), those get documented and rubber-stamped by our change control process, but beyond that it's assumed that regular patches are approved.
At one point it was suggested that for compliance I was going to have to not only document and justify all patches applied, but pre-document exactly which patches were being applied on each system AND show proof that those patches and ONLY those patches were applied on each system. Given the way that RHEL releases patches (not to mention Debian and Ubuntu, that would have turned a monthly patch cycle that takes at most a day or two into a full-time job. I pushed back, pointing out that it would do nothing other than create more paperwork and take time away from my ability to do anything else, and they eventually agreed. Yes, there can be a happy medium somewhere, and it's not the same for every organization, but the fewer technical people there are doing the actual work, the less this type of bureaucracy will produce functional results...
Re:Setup a WSUS server (Score:2, Informative)
Also, get the CAB to set policy to pre-approval any patches marked "security" or "critical". This can be configured within WSUS. That will cut the workload/paperwork down significantly.