Disempowering the Singular Sysadmin? 433
An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"
Too many cooks... (Score:0, Interesting)
...spoil the soup.
how do they design nuclear missile systems? (Score:5, Interesting)
look at programs where there is a lot of technical activity and communication activity for time sensitive work
you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive
so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins
and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay
Reinventing history (Score:5, Interesting)
would be enormously cumbersome if they needed a consensus of administrators to execute.
Thats why you leave changes to the 24x7 onsite operations team not one lone admin doin' his thing in the cube. They're the ones monitoring the systems, seems most sensible if they "push the buttons" on the things they watch. Ideally you have one team that does nothing but watch and one team that does nothing but do, and theoretically they cooperate.
And besides, DAN appears still to be vaporware.
DAN appears to be a poor reinvention of flight control software for aerospace from the 70s/80s. Those whom don't know their history are doomed to poorly repeating their past.
Next up, we'll reinvent the concept of the security office from AS/400, or maybe the idea of hard realtime control.
Maybe someone out there could could reinvent the concept of the watchdog timer so the "DAN" cluster doesn't go into deadlock? Naah, we'll let them "discover" it themselves, the hard way.
Re:sternobread (Score:5, Interesting)
That is how all of our servers are setup. I'm just a "developer" that uses them but I believe no one knows the root password for our systems. It is a *big* random string that is printed out by the sysadmin that sets up the machine, sealed in an envelope with that person's signature on both sides and stuck in a safe. In the event that a machine is so hosed that the root password is needed it is used and then a new one is generated and sealed away again.
Everyone uses sudo for everything. All sudo access is logged.
The system isn't perfect of course, nothing is, but it goes a long way to the worry of one person having root keys for things.
It's not your only line of defense (Score:4, Interesting)
First, understand that Slashdot is only going to provide a hint of what you will be doing. Security is complex and easy to get wrong, and there's a whole lot of evidence of that in the news. If security is important to your company, you should invest in a CISSP to really help you get things set up in a fashion that the industry considers to be best practices. Until then, consider these few generic suggestions.
Multiple layers of security help ensure that nothing goes astray, or if it does that it's detected before too much damage is done. And separation of duties helps make sure that one rogue actor can't do it all by himself.
Separate the admin of the box from the admin of the data. The guy who holds the root PW doesn't have to be the same guy who holds the private key for the database.
Add off-the-box auditing to the actions of root. As soon as someone signs on as root, notification is sent to a different box of the originating IP and it's timestamped. Don't let your application sysadmin be the sysadmin of the audit box! And the auditor should investigate carefully any situations that are out of the ordinary. (This box fell off the network just before root logged on? That's an odd coincidence.)
Define expected behavior with policies. If you want to run a trustworthy ship, clearly stating who has access to do what with which systems eliminates confusion, and helps avoid where one sysadmin creeps over into other systems.
Ultimately, you've placed trust your admin to do a job, and you need to trust him or her to do that job. Somebody's got to be root. But they also have to know they'll be held accountable for what they do.
The Orange Book solution (Score:4, Interesting)
This is an old problem in high assurance systems. As other posters have pointed out, as some point you have to trust someone. But you can still "trust but verify".
The standard solution is "division of privilege". Over time folks have learned that the key is a system which audits everything the admin does, and the one thing the admin can't do is modify or delete the audit trail. A separate person or team has the role of auditor.
This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs. The solution there is trust someone, and be ready to fire, sue, and/or prosecute if they violate that trust.
Re:Yes (Score:5, Interesting)
A subset of administrative applications requiring multiple administrators may not be such a bad compromise.
ex:
* change root password (or password to any "wheel" account) - requires multiple administrators to enter the same passwords
*su/sudo'ing to a "wheel" account, or changing said account's privileges, requires the authorization of at least one other wheel'ed user.
* Alterning an active network interface, shutting down, and restarting requires authorization by other administrative users.
Stuff like that, which are things that shouldn't be done often, anyway, and could allow one admin to take over the whole system, seem like good candidates for multiple-approvals. Everything else could be left alone.
The approval process is basically - the root users needs to take the action, and then 2+ non-root (but wheel) users must approve it.
I'm using 'wheel' as that is the group in FreeBSD that is typically allowed access to sudo/su. Not sure how other systems typically work.
Re:Answer in kdawson's tagline (Score:4, Interesting)
"trust but verify"
To get some transparency / accountability, just set up an authlog black hole that includes all of the sudo activity from your servers.
Re:There is a well tested method for that (Score:5, Interesting)
Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?). Your plans are torn apart and put back together. Departmental announcements, customer notifications, etc, etc. Accounting wants numbers, and proposals from 3 sources for the cost of a replacement drive (which you have 5 of in the datacenter, and a regular supplier). You're sitting there with the mind numbing noise flowing past. All you can think is "the array was set up with no hot spare. It's running in a degraded mode. Change the damned drive." Of course, complaints of slow drive performance are scattered throughout the meeting.
Two months and more meetings than you can remember later, they slate it for an arbitrary windows. Saturday at 3am. Not only change it, but you are required to stay while it rebuilds, "just in case...". Just in case? You have me working 8 to 7 Monday through Friday, weekends on demand (which are every weekend) AND you want me to blow off Saturday night to do the change? Ah who cares, I don't need sleep.
Then Thursday afternoon before the schedule change is done, a second drive in the array fails, and the whole thing is down. All the same people who were in on the meetings start screaming "How could you let this happen?!"
Thursday afternoon becomes Thursday night, and by Friday morning you have the array back up and working, through some dumb luck. (crossing fingers, praying to whatever gods may be listening, and tapping the drive with a screwdriver at boot time to make it spin up). The only planning that helped is that you keep a change of clothes and a toothbrush in the car, since you don't have time to go home once you're done. In doing the work, you notice the same thing happening to a neighboring machine. Damned aging hardware. So you just change it without the mess that accompanied the first change. Not only are you bitched out for not fixing the first array in time, but you get it twice as bad for fixing the other one before it became a problem. How could you have independent thought? How could you make a change without proper authorization?
The only thoughts still in your head are "I hate this job", "my car keys are in my pocket, and I could just leave." Is this the day you quit? Maybe, just maybe. Just one more thing, and that'll be it. I don't need this shit.
Friday afternoon, not sleeping since Wednesday night, you are told "Do [some other task] after hours tonight." No, you won't get paid any overtime since you're on salary. The task will take at least 8 hours, and they need it done before Saturday morning. Do you scratch out a resignation with a sharpie on the CEO's wall at 2am, or do you just walk out?
I really hated that job.
Re:Too many cooks... (Score:5, Interesting)
The submission seems to presume that the system in question is some sort of *nix or Windows box. If we look into the world of mainframe operating systems, we'll see that this has already been fully adressed, and any number of individuals with discrete UIDs may have superuser access. This has evolved out of a history where sysadmins worked shifts, so sharing a single privileged UID/password was/is a bad idea.
The way such access is administrated needs a proper policy within the organisation, though. Back in the '90s, I worked at one outfit (an insurance company) where the vice-CEO demanded superuser privileges despite having no knowledge of system administration or any other computing background. He just wanted to act as overlord as to what staff had access to on their signons. I was very tempted to tell him to get fucked, phrased in more professional terms. Like "Go get professionally fucked".
My immediate boss was (wisely) more inclined to a diplomatic approach, however, so he pursuaded me to install a dummy program for him that was enough to convince him that he had what he wanted, without granting him any kind of command line access, or ability to change system configuration.
Re:There is a well tested method for that (Score:5, Interesting)
Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?).
Not that I don't agree that some companies make change management more than it needs to be (mine does it OKAY), but I bet the guy I knew years ago who changed a drive on a RAID-5 array had thought about testing and rollback. You see, he received the replacement drive late in the day, ran into the data center, popped out a drive, popped in the new drive, and went home. Sadly, he had pulled the wrong drive.