Disempowering the Singular Sysadmin? 433
An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"
In other news... (Score:5, Insightful)
Rule by a benevolent dictator has certain advantages, and rule by committee has certain opposite advantages. It was ever thus.
There is a well tested method for that (Score:5, Insightful)
It is called: "Change Control" and usually goes along with "Revision Control" on configs.
If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.
Re:respect (Score:2, Insightful)
It isn't about respect, necessarily. I am a sysadmin that has the keys to a lot of things and I have wondered about this very problem. It isn't about how much respect I deserve but it would be nice to a have a distributed method in the event of some sort of catastrophe or something as simple as being sick.
Re:There is a well tested method for that (Score:5, Insightful)
There's a reason.. (Score:4, Insightful)
That you have one person doing it. It's effective, and versatile.
If you have multiple people empowered to do exactly the same thing, you end up at the mercy of the one that decides to shut everyone else out.
If you then have a security admin that's the only one to be able to alter the login info, then you're at their mercy.
With the "dual key" type approach, what's to stop someone installing a back door along with a normal software upgrade? Does everyone have the same knowledge as your prime sysop? Can you afford to have one person that completely mirrors another, instead of distributing the skills across a time (with duplication covered across the team)?
What if both the key holders are in cahoots?
Interestingly, who is stopping your CEO from making those really bad decisions, or your FD from siphoning the cash, or a whole host of other areas where you trust one person to do a job?
Value the person, and make sure you treat them well enough to make it not worth their while to play you up.. Then you'll have no problem.
Screw them over at every opportunity, and you'll really have to trust their ethical views (you're still usually safe, but it's no guarantee then).
There are Safeguards Already (Score:5, Insightful)
Mostly, except in very small organizations, there are several implicit safeguards to keep any one person from doing evil with the systems. They are subtle, but effective.
Peer review: Most sysadmins are hired by other sysadmins, or at the very least a technical manager. This means that you are hired based on your skills, reputation, track record, and demonstrated attitude. This means that ideally, you wouldn't even *think* about intentionally subverting a system, because that would mean breaking it or compromising it in some way, and most professional SA'a are simply too OCD to allow it.
Business continuity: Most organizations have several layers of continuity in place, such as disaster recovery scenarios, system snapshots, monitoring, and auditing. This means that unless you are VERY subtle, or work for an entirely incompetent team, you WILL get caught, and the damage will be minimized as you are being put into a police car, never to work in IT again.
There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.
No technical solution will ever replace good management and planning, and a design that eliminates the vulnerabilities of a system to rogue sysadmins, will also eliminate its flexibility. It's just a lot cheaper and easier to try and run a good shop.
Re:There is a well tested method for that (Score:5, Insightful)
Works, although excruciatingly slowly for planned work.
The collision of excruciatingly slow proactive planned work, and reactive trouble tickets, always is a source of utter hilarity. Usually the end result is you only do planned proactive paper shuffling for meaningless stuff "lets change the background color to be 0.001% darker" and ram thru development as part of a trouble ticket with no oversight at all (well, to make our big customer happy, we've decided to completely redo our database schema and stored procedures this afternoon as part of the ticket).
Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maint, and a half hour to replace a failed drive at any time, this simply eliminates all proactive maintenance. Much easier / cheaper to burn the power supply out, have a nice long outage, and then replace the whole device, than to get permission to blow dust out of the air filter.
The end result is usually much worse than it was at the beginning.
How about (Score:4, Insightful)
Also, simple change management alleviates most of these problems. Even if it's just a log for what happened so that the next shift or your colleague tomorrow knows what you did today. Then again, I guess that is really back to acting like adults.
Re:why? (Score:5, Insightful)
Not really. It's fun to think I could do anything I wanted, but I don't want to. I like my job, I like the people I work with, I don't want to screw them over. It's nice to have an employer that trusts you too. If I wasn't trusted, I would probably just leave. If they want me to be able to administer and troubleshoot everything, I obviously need full access.
Smack * (Score:4, Insightful)
Peer Review, Change Control, Auditing, Maintenance Windows, Testing all changes in a lab before production, source and version control / maintenance. These are all best practices, work regardless of operating system and don't require any special software.
Why o why do you want to use software to take the place of established best practices? Best practices are there for good reasons, and those reasons usually have multi-million dollar lessons attached to them. You don't need special software, just a heavy that says yes you /must/ do it this way and raises hell when you try otherwise...
Re:why? (Score:4, Insightful)
if you can't trust the person at the top: then either they don't deserve to be there, or you need to find a new job.
when you're the person at the top: you better have earned the trust and respect of those under you. Subverting it does nobody any good in any long term.
Re:respect (Score:3, Insightful)
You keep your passwords in a network share? Are you schizophrenic or just incompetent?
I hope that file is fucking well encrypted ... but even in that case, it's just a bad idea.
Re:There's a reason.. (Score:4, Insightful)
who is stopping your CEO from making those really bad decisions
The board; other executive officers, and limitations for class of big decisions that requite a vote of shareholders; (especially in non-public companies)
or your FD from siphoning the cash,
Periodic independent audit, as well as requirement of extra authorisation for amounts above X - in any well managed company FD can't siphon all cash without other officers getting dirty as well;
or a whole host of other areas where you trust one person to do a job?
There are no other areas where high-risk issues are trusted to one person without serious oversight. In most companies the IT management and auditing is either solved as well, or the only remaining weak point with this problem - that's why the article is there.
Valuing persons and treating them well is in no way a solution - compare 'security by obscurity' vs. 'security by goodwill' vs. 'security by prayer' and you'll find some similarities.
Four-eyes principle stops a lot of potential malice, as the likelihood of both keyholders being ethically faulty and not betraying each other is much, much lower than simple chance of one person being ethically faulty.
Installation of back doors along with a normal software upgrade is a prime reason why someone other than 'your prime sysop' needs to periodically verify stuff; if you don't mirror, then you ask for outside audit of stuff; have secure write-only logging of 'root' tasks to a system which is completely controlled by someone else, etc.
Of course, it depends on the risks - if the worst your sysadmin can do is shut down an informative website that you have, then it's no big deal. If it's a payment system that can fund a life-long vacation in the Bahama's for an opportunistic administrator, then we're talking about all such measures.
Re:Yes (Score:5, Insightful)
Re:sternobread (Score:5, Insightful)
The solution here is to follow a reasonable security protocol in writing the sudoers file. Specifically, the default action is to prohibit. Permitted actions are then whitelisted. On a high-security system, no entry should allow a user to sudo su -. Problem solved.
Incidentally, I see no point in locking down users who have physical access to the DC.
Re:Yes (Score:2, Insightful)
Trust, but verify. I believe the submitter is asking how to provide for verification without breaking operations.
Just as I'd be an idiot for handing my checkbook over to the sole control of an employee based solely on trust, I'd be an idiot for handing over the keys to my IT systems.
Re:why? (Score:4, Insightful)
Obviously there are those with different goals and standards and it's not always easy to identify them. I'm not sure how to prevent that -- someone who over the years gradually gets more access and one day they decide to go rogue and do something harmful. Even minimizing the attack surface you usually have that single admin account that owns everything else. Maybe I should read the article.
Re:Yes (Score:4, Insightful)
Re:sternobread (Score:4, Insightful)
I think you're missing the point. Auditing/logging systems are not meant to provide effective defense. They are meant to let PHB's mark appropriate check boxes on compliance forms and sleep better without worrying what those evil nasty sysadmins are doing. Don't confuse them.