Disempowering the Singular Sysadmin? 433
An anonymous reader writes "Practically every computer system appears to be at the mercy of at least one individual who holds root (or whatever other superuser identity can destroy or subvert that system). However, making a system require multiple individuals for any root operation (think of the classic two-key process to launch a nuke) has shortcomings: simple operations sometimes require root, and would be enormously cumbersome if they needed a consensus of administrators to execute. There is the idea of a Distributed Administration Network, which is like a cluster of independently administered servers, but this is a limited case for deployment of certain applications. And besides, DAN appears still to be vaporware. Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?"
Well... (Score:2)
We just have an account in Active Directory called "Support" with Administrative Rights across the domain, and the Sysadmin holds the Root and Administrative Passwords to effectively hold total control if need be. He can change the Support Password and lock all of us out if he wishes, or give us the info to let us back in. But pretty much anything that needs to be done, we can do on that account, including adding the PC to the domain itself.
Or am I going to be laughed at for posting the Microsoft answer?
Re: (Score:2)
Perhaps a stupid question, but if you're in the Administratos group, can't you change the Administrator password anyway?
Probably he's changed the security permissions for that account, I just am interested!
Re: (Score:2)
It's placed in a Container we don't have access to. In essence we can't touch that account.
Re: (Score:2)
Or you could add the people who need admin rights to the appropriate group, which is a lot more secure because you can then audit what individuals do. A general-use user account with that level of privilege is generally considered a Very Bad Idea unless you really can't avoid it.
Re: (Score:2)
I would generally tend to agree, in the growing pains of the company - they hired more lab people then they had computer techs to handle, as such a few labs have generic lab accounts, and whenever one of them gets a virus or deletes something important by accident or whatever, there's no one you can point the finger at because there's no log besides the generic account.
And the same thing is open with our support section, anonymity with admin rights, very dangerous should someone screw up or should someone w
Re: (Score:2)
no, but you'll be laughed at for having a single shared account that means anyone who logs in to perform some "support activity" (maybe after a few drinks, or just general brainfarts) cannot be determined after the event. This can be a good thing, depending on how bad your admins are (good for them, that is). :)
In other news... (Score:5, Insightful)
Rule by a benevolent dictator has certain advantages, and rule by committee has certain opposite advantages. It was ever thus.
more people means more auditing (Score:2)
we have a few databases where selected developers can do anything they want since they do most of the work there and there is no SOX requirement for those databases. every week mysterious things happen where column schemas are changed, stored procedures are updated, etc with no notification to anyone except when trouble tickets come in because some other application broke
There is a well tested method for that (Score:5, Insightful)
It is called: "Change Control" and usually goes along with "Revision Control" on configs.
If you change without recording the reason for change and without checking in the result so that the two versions can be compared and analysed you get a pink slip. Voila. Problem solved.
Re:There is a well tested method for that (Score:5, Insightful)
Re: (Score:2)
we do this - there's a spreadsheet (well, got to have something right) that gets updated whenever anyone dials in to a customer site, even if you did nothing. If you do something when dialled in, that gets logged too.
I don't know if failing to update it is a pink-slippable offence, but you will get a severe b*ll*cking if you fail to do it twice.
BTW, our customers are police, fire and ambulance control centres. Maybe that makes us different to the usual, but it's simple and works well. If we could get rid of
Re: (Score:3)
We do it, a major publicly traded international company with thousands of employees. Its not hard: make /etc a repository (we like mercurial), have puppet manage your servers, and revision control the server config files on the puppetmaster (again, mercurial helps here).
Re: (Score:2)
Doesn't solve the stupid admin from logging in to my server and entering "reboot". Which is more of a problem in my case then configuration files (they won't even touch those with a 10 foot pole)
Re: (Score:2)
It may not solve the problem, but the pink slip will solve recurrences of the problem.
Re: (Score:3)
and how many people are always changing minor things without change control because they feel this is their baby and they can do anything?
Re: (Score:2)
and how many people are always changing minor things without change control because they feel this is their baby and they can do anything?
That's because the development server IS the production server, for whatever reasons. Its not a maintenance procedure problem, but a design problem way upstream of scheduled maintenance.
The other scenario is when you're breaking individual (or world wide) new ground. It works when a huge team can spend months debating the route and design for some new railroad tracks, however an operating engineer needs full and instant discretion of how and when to work the throttle and brake levers.
There's some things t
Re: (Score:2)
and unless you break something really bad or cause a big problem with a SOX audit no one is going to care since it costs $30,000 or so to hire a new person
Re:There is a well tested method for that (Score:5, Insightful)
Works, although excruciatingly slowly for planned work.
The collision of excruciatingly slow proactive planned work, and reactive trouble tickets, always is a source of utter hilarity. Usually the end result is you only do planned proactive paper shuffling for meaningless stuff "lets change the background color to be 0.001% darker" and ram thru development as part of a trouble ticket with no oversight at all (well, to make our big customer happy, we've decided to completely redo our database schema and stored procedures this afternoon as part of the ticket).
Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maint, and a half hour to replace a failed drive at any time, this simply eliminates all proactive maintenance. Much easier / cheaper to burn the power supply out, have a nice long outage, and then replace the whole device, than to get permission to blow dust out of the air filter.
The end result is usually much worse than it was at the beginning.
Re: (Score:2)
Another example, if it takes a month and endless meetings to replace a failing drive during scheduled maint, and a half hour to replace a failed drive at any time, this simply eliminates all proactive maintenance. Much easier / cheaper to burn the power supply out, have a nice long outage, and then replace the whole device, than to get permission to blow dust out of the air filter.
The end result is usually much worse than it was at the beginning.
Of course the first one follows all ITIL processes
How long the corporate world will see that CMMI and ITIL are the (very expensive) equivalent of 'power crystals' and astrology?! Oh yeah, that's right, NEVER
Re:There is a well tested method for that (Score:5, Interesting)
Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?). Your plans are torn apart and put back together. Departmental announcements, customer notifications, etc, etc. Accounting wants numbers, and proposals from 3 sources for the cost of a replacement drive (which you have 5 of in the datacenter, and a regular supplier). You're sitting there with the mind numbing noise flowing past. All you can think is "the array was set up with no hot spare. It's running in a degraded mode. Change the damned drive." Of course, complaints of slow drive performance are scattered throughout the meeting.
Two months and more meetings than you can remember later, they slate it for an arbitrary windows. Saturday at 3am. Not only change it, but you are required to stay while it rebuilds, "just in case...". Just in case? You have me working 8 to 7 Monday through Friday, weekends on demand (which are every weekend) AND you want me to blow off Saturday night to do the change? Ah who cares, I don't need sleep.
Then Thursday afternoon before the schedule change is done, a second drive in the array fails, and the whole thing is down. All the same people who were in on the meetings start screaming "How could you let this happen?!"
Thursday afternoon becomes Thursday night, and by Friday morning you have the array back up and working, through some dumb luck. (crossing fingers, praying to whatever gods may be listening, and tapping the drive with a screwdriver at boot time to make it spin up). The only planning that helped is that you keep a change of clothes and a toothbrush in the car, since you don't have time to go home once you're done. In doing the work, you notice the same thing happening to a neighboring machine. Damned aging hardware. So you just change it without the mess that accompanied the first change. Not only are you bitched out for not fixing the first array in time, but you get it twice as bad for fixing the other one before it became a problem. How could you have independent thought? How could you make a change without proper authorization?
The only thoughts still in your head are "I hate this job", "my car keys are in my pocket, and I could just leave." Is this the day you quit? Maybe, just maybe. Just one more thing, and that'll be it. I don't need this shit.
Friday afternoon, not sleeping since Wednesday night, you are told "Do [some other task] after hours tonight." No, you won't get paid any overtime since you're on salary. The task will take at least 8 hours, and they need it done before Saturday morning. Do you scratch out a resignation with a sharpie on the CEO's wall at 2am, or do you just walk out?
I really hated that job.
Re:There is a well tested method for that (Score:5, Interesting)
Sadly enough, I've had a simple drive replacement tied up in meetings and other office politics for months. Write up a proposal for change, sit in meetings where various department heads without a clue discuss the potential hazards, write up the rollback process (for changing a drive?).
Not that I don't agree that some companies make change management more than it needs to be (mine does it OKAY), but I bet the guy I knew years ago who changed a drive on a RAID-5 array had thought about testing and rollback. You see, he received the replacement drive late in the day, ran into the data center, popped out a drive, popped in the new drive, and went home. Sadly, he had pulled the wrong drive.
Re: (Score:3)
hehe.
Sorry, I had to laugh.
That has more to do with check your work than it does with the prolonged control processes that businesses put in place. I've seen the control processes made by committee fail miserably. Sure, they want all this stuff done. The 400 point checklist frequently misses some essential piece, like "is it the right drive?" and "verify it's rebuilding properly". The last step would have screamed "You're doing it wrong".
If there's no clear
Re: (Score:3)
Once a day, we had a change to do and this change need to be coordinate with a mainframe change. We were there to test the procedure on test environment, with the test mainframe partition. In the operators' room, there is three levels of desk, each level seeing what is going on on the other level. It was very like the Star Trek Enterprise command room or someth
If the single SysAdmin is even half decent.. (Score:2)
Re: (Score:3)
And the top programs run through sudo? "sudo su" and "sudo sh" :)
The article wasn't suggesting controls for a single admin to accomplish a task. They were talking about requiring at least 3 admins to do the same thing in three identical environments to accomplish one task.
"Ok, we need to reboot server X, all of you on my mark type 'shutdown -r now' ... 3 ... 2 ... 1 ... mark"
"Dammit Mark, you didn't hit enter in time. Lets try again."
how do they design nuclear missile systems? (Score:5, Interesting)
look at programs where there is a lot of technical activity and communication activity for time sensitive work
you can't have a nuclear missile system where one guy can invoke the bombs to go off. at the same time, the system has to be quick and responsive
so you need to engineer administrative systems where not less people are involved but MORE: you can't do this function or that function without also involving this guy over there turning a key, etc.: all admin functions invoke more than one person. that's the best way to have a system where power can't be abused. its about redundancy and layers of admins, not less admins
and if people are pursuing this question because they don't want to pay an admin or can't trust someone else with their system, then such idiots get the system they deserve: a broken one and no one willing to fix it at the money you want to pay
yeah (Score:2)
Eventually, you have to trust someone. (Score:5, Funny)
Oh, the jobs people work at! Out west, near Hawtch-Hawtch, there's a Hawtch-Hawtcher Bee-Watcher. His job is to watch... is to keep both his eyes on the lazy town bee. A bee that is watched will work harder, you see.
Well...he watched and he watched. But, in spite of his watch, that bee didn't work any harder. Not mawtch.
So then somebody said, 'Our old bee-watching man just isn't bee-watching as hard as he can. He ought to be watched by another Hawtch-Hawtcher! The thing that we need is a Bee-Watcher -Watcher!'
Well... The Bee-Watcher-Watcher watched the Bee-Watcher. He didn't watch well. So another Hawtch-Hawtcher had to come in as a Watch Watcher-Watcher!
And today all the Hawtchers who live in Hawtch-Hawtch are watching on Watch-Watcher-Watchering-Watch, Watch-Watching the Watcher who's watching that bee.
You're not a Hawtch-Watcher. You're lucky, you see.
Re: (Score:2)
Reinventing history (Score:5, Interesting)
would be enormously cumbersome if they needed a consensus of administrators to execute.
Thats why you leave changes to the 24x7 onsite operations team not one lone admin doin' his thing in the cube. They're the ones monitoring the systems, seems most sensible if they "push the buttons" on the things they watch. Ideally you have one team that does nothing but watch and one team that does nothing but do, and theoretically they cooperate.
And besides, DAN appears still to be vaporware.
DAN appears to be a poor reinvention of flight control software for aerospace from the 70s/80s. Those whom don't know their history are doomed to poorly repeating their past.
Next up, we'll reinvent the concept of the security office from AS/400, or maybe the idea of hard realtime control.
Maybe someone out there could could reinvent the concept of the watchdog timer so the "DAN" cluster doesn't go into deadlock? Naah, we'll let them "discover" it themselves, the hard way.
There's a reason.. (Score:4, Insightful)
That you have one person doing it. It's effective, and versatile.
If you have multiple people empowered to do exactly the same thing, you end up at the mercy of the one that decides to shut everyone else out.
If you then have a security admin that's the only one to be able to alter the login info, then you're at their mercy.
With the "dual key" type approach, what's to stop someone installing a back door along with a normal software upgrade? Does everyone have the same knowledge as your prime sysop? Can you afford to have one person that completely mirrors another, instead of distributing the skills across a time (with duplication covered across the team)?
What if both the key holders are in cahoots?
Interestingly, who is stopping your CEO from making those really bad decisions, or your FD from siphoning the cash, or a whole host of other areas where you trust one person to do a job?
Value the person, and make sure you treat them well enough to make it not worth their while to play you up.. Then you'll have no problem.
Screw them over at every opportunity, and you'll really have to trust their ethical views (you're still usually safe, but it's no guarantee then).
Re: (Score:3)
But ... make sure you have a backup in case the person gets hit by a bus.
Re: (Score:2)
Re:There's a reason.. (Score:4, Insightful)
who is stopping your CEO from making those really bad decisions
The board; other executive officers, and limitations for class of big decisions that requite a vote of shareholders; (especially in non-public companies)
or your FD from siphoning the cash,
Periodic independent audit, as well as requirement of extra authorisation for amounts above X - in any well managed company FD can't siphon all cash without other officers getting dirty as well;
or a whole host of other areas where you trust one person to do a job?
There are no other areas where high-risk issues are trusted to one person without serious oversight. In most companies the IT management and auditing is either solved as well, or the only remaining weak point with this problem - that's why the article is there.
Valuing persons and treating them well is in no way a solution - compare 'security by obscurity' vs. 'security by goodwill' vs. 'security by prayer' and you'll find some similarities.
Four-eyes principle stops a lot of potential malice, as the likelihood of both keyholders being ethically faulty and not betraying each other is much, much lower than simple chance of one person being ethically faulty.
Installation of back doors along with a normal software upgrade is a prime reason why someone other than 'your prime sysop' needs to periodically verify stuff; if you don't mirror, then you ask for outside audit of stuff; have secure write-only logging of 'root' tasks to a system which is completely controlled by someone else, etc.
Of course, it depends on the risks - if the worst your sysadmin can do is shut down an informative website that you have, then it's no big deal. If it's a payment system that can fund a life-long vacation in the Bahama's for an opportunistic administrator, then we're talking about all such measures.
Re: (Score:2)
> Interestingly, who is stopping your CEO from making those
> really bad decisions, or your FD from siphoning the cash,
> or a whole host of other areas where you trust one person to do a job?
Nothing. Sometimes it is the CxO that is making some clueless change without telling anyone that subsequently breaks everything.
Audit (Score:2)
Really though if they have physical access to something they can do whatever they like. Auditing and logs can go pretty far but at some point you have to trust the people that run things.
There are Safeguards Already (Score:5, Insightful)
Mostly, except in very small organizations, there are several implicit safeguards to keep any one person from doing evil with the systems. They are subtle, but effective.
Peer review: Most sysadmins are hired by other sysadmins, or at the very least a technical manager. This means that you are hired based on your skills, reputation, track record, and demonstrated attitude. This means that ideally, you wouldn't even *think* about intentionally subverting a system, because that would mean breaking it or compromising it in some way, and most professional SA'a are simply too OCD to allow it.
Business continuity: Most organizations have several layers of continuity in place, such as disaster recovery scenarios, system snapshots, monitoring, and auditing. This means that unless you are VERY subtle, or work for an entirely incompetent team, you WILL get caught, and the damage will be minimized as you are being put into a police car, never to work in IT again.
There are no "indispensable people:" If you are a sysadmin, and you are the only one who knows your systems, you have not done your job. Every system and app should be documented, and there should be accountability for every change and decision.
No technical solution will ever replace good management and planning, and a design that eliminates the vulnerabilities of a system to rogue sysadmins, will also eliminate its flexibility. It's just a lot cheaper and easier to try and run a good shop.
You need at least TWO good sysadmins... (Score:3)
Hire admins who know their stuff and make sure you have at least two of them with the root password. Make sure they've got some kind of change control in place, and make sure you have them document what they're doing.
I've been the sole sysadmin before, and I always felt worried that my legacy, should I be fired or quit or hit by a bus, would be "She didn't do a great job because everything fell apart after she left/was fired/was bussassinated". So, I always tried to document things and made sure the boss had the "keys to the kingdom" (document with root pw and locations of my documentation to give to my successor).
Re: (Score:2, Funny)
bussassinated
I have a new word of the day! Thank you. :D
Re: (Score:3)
bussassinated has 2 possible meanings
1) killed by bus
2) killed by business
It's not your only line of defense (Score:4, Interesting)
First, understand that Slashdot is only going to provide a hint of what you will be doing. Security is complex and easy to get wrong, and there's a whole lot of evidence of that in the news. If security is important to your company, you should invest in a CISSP to really help you get things set up in a fashion that the industry considers to be best practices. Until then, consider these few generic suggestions.
Multiple layers of security help ensure that nothing goes astray, or if it does that it's detected before too much damage is done. And separation of duties helps make sure that one rogue actor can't do it all by himself.
Separate the admin of the box from the admin of the data. The guy who holds the root PW doesn't have to be the same guy who holds the private key for the database.
Add off-the-box auditing to the actions of root. As soon as someone signs on as root, notification is sent to a different box of the originating IP and it's timestamped. Don't let your application sysadmin be the sysadmin of the audit box! And the auditor should investigate carefully any situations that are out of the ordinary. (This box fell off the network just before root logged on? That's an odd coincidence.)
Define expected behavior with policies. If you want to run a trustworthy ship, clearly stating who has access to do what with which systems eliminates confusion, and helps avoid where one sysadmin creeps over into other systems.
Ultimately, you've placed trust your admin to do a job, and you need to trust him or her to do that job. Somebody's got to be root. But they also have to know they'll be held accountable for what they do.
The Scientology approach (Score:2)
Powerbroker & logging (Score:5, Informative)
We have several solutions which work together to minimize the risk of root at my company:
1. Powerbroker. It's in use on every single UNIX system administered by our Global IT teams. Every user has a role (or several roles), and that allows them to execute a variety of commands with elevated privileges. Once Powerbroker is invoked, however, every single keystroke is logged and can be played back. These logs are stored indefinitely; access is very restricted.
2. Automated, centralized root password management. One of the steps to setting up a UNIX machine here is ensuring the root password and remote console admin passwords match that dictated by our automated provisioning system. Then every 30-90 days (depending on policy for this type of system) the root password is changed to a very long, apparently very random string. I can look this password up if my role allows it, but the lookup is also logged.
3. A good Change Request (CR) process. Every system that exists in a data center should have a record in our systems database. Once a system has passed through the phases of deployment (Warehouse -> Data Center Install -> Sysadm Configure -> Deployed) any change made to the system must be requested and approved by the owners of the system. This approval is logged, and the date/time of the work is also logged. Sysadms must close service requests within the time window specified by the CR, or apply for an extension or reschedule if they're unable to complete it within the allotted time.
The downside to this is that you lose quite a bit of system administrator work hours filing and managing change requests. However, this loss of efficiency -- IMHO -- is better than the mayhem that ensues without an organized change process.
4. Automated forensic tools to monitor changes. Information overload is a real risk with any Tripwire-style system, though. We're still working out some of the kinks on this part of the system. Once we ensure that all normal changes due to operation of the system and scheduled maintenance get excluded, this will be the fourth leg to reduce the risk of super-user privileges.
At any company, IT must find a balance between controlling user actions and monitoring those actions. In most cases, the easiest approach is to prohibit by policy only those things that might typically result in lawsuits, but monitor everything else to the best of your ability. Combining a Powerbroker-like product with automated root password management -- both with fascistic logging -- is a reasonable approach that works well for many large companies. Combine this with a change management system, and a forensic tool to automatically monitor and notify of unauthorized changes, and super-user isn't really all that big of a concern.
Re: (Score:3)
These logs are stored indefinitely; access is very restricted.
to whom? what you have to keep in mind is that computers operate as single minded entities. when you approach a machine like that: security is currently an afterthought. this tells me that there is somebody that holds access above the other users, basically missing the point here.
I can look this password up if my role allows it, but the lookup is also logged
Again, that means that there's somebody administering the logging system. and I almost assure you that even if their logins are listed somewhere: they have full access to remove those entries and make it look like it never happen
Re:Powerbroker & logging (Score:5, Informative)
You've tossed out a few red herrings and a couple of valid points. I'll try to address them in order.
No, I haven't missed the point at all. The point is to distribute the responsibility with sufficient checks in order to ensure that misbehavior will be caught and dealt with in a timely fashion. Is it possible someone could scheme up a way to slide abuses past the admins? Of course it is. But between good backups, fascistic logging, role-based access control, and routine audits by the change control committee, the risk is minimized.
There's no one person who holds the "keys to the kingdom". No critical data is stored on the machines themselves; it's all stored on centralized storage. The folks who admin the automated root password changes don't have any access to storage; the storage folks typically don't have any access to the systems.
Incorrect. I didn't cover this in my original post, but logs are (and should be) stored on write-once media. You can designate volumes on modern storage media so that, once written, it can never be altered without destroying the entire volume. We use this extensively.
Once again, distributed access can prevent this. The network team and the sysadm team aren't the same teams. Every port on your switch is disabled until it's enabled by the network team. Even once enabled, that port must be on the same VLAN as the hypothetical credit-card storage system.
That's once again where fascistic logging and automated reporting come into play. If a port is disconnected, unless a host has been blacked out with an appropriate change control ticket filed, the port disconnection generates an immediate Priority 1 service request to investigate.
If a drive is removed from centralized storage, that also generates an immediate P1 ticket. The sysadm's access would have been logged the moment he swiped his badge, and cameras throughout the data center capture the switch-over.
A corrupt admin can do a lot of damage, I admit. There's no getting around it. But with sufficient logging -- and yes, I include physical surveillance as "logging" too -- they're not going to get away with it.
Now this is the red herring. If you've ever done ANYTHING major with credit cards in a data center, you are aware that you're subject to yearly audits of your infrastructure by Payment Services. They do a deep-dive of your systems to enforce a huge number of requirements. I can't go into it here. It literally fills a large book, and they go over it line-by-line with all the admins involved, every single year. I've been through several of these, and each year it gets broadened to cover more potential issues.
Chief among these requirements? A separate admin/management network from the front-end/back-end network. You can't "tunnel in" to that network and make it "act like" another system. The network is an unroutable private VLAN or fibre-channel connection.
Ye
Comment removed (Score:3)
Driving a car (Score:3)
The only solution I can think of that would stand a chance is to require:
a) everything gets documented (you'll know this is the correct way, as all the techies will hate it)
b.) every week / month all the roles change, if an admin coming into a role finds that things aren't as they were documented, someone gets yelled at
This also has the advantage that you're no longer completely screwed if someone leaves, goes sick or gets promoted. it also makes it clear to the people in question that the company can get along quite nicely without them.
How about (Score:4, Insightful)
Also, simple change management alleviates most of these problems. Even if it's just a log for what happened so that the next shift or your colleague tomorrow knows what you did today. Then again, I guess that is really back to acting like adults.
The Orange Book solution (Score:4, Interesting)
This is an old problem in high assurance systems. As other posters have pointed out, as some point you have to trust someone. But you can still "trust but verify".
The standard solution is "division of privilege". Over time folks have learned that the key is a system which audits everything the admin does, and the one thing the admin can't do is modify or delete the audit trail. A separate person or team has the role of auditor.
This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs. The solution there is trust someone, and be ready to fire, sue, and/or prosecute if they violate that trust.
Re:The Orange Book solution (Score:4, Informative)
This is one of the requirements of a B2 level system in the old Orange Book model, and you'll see if it as a requirement if you need to provide systems for most countries' military or intelligence organizations. It's rarely used elsewhere because more or less noone else is willing to pay the staffing costs.
Right. I developed an OS for that model many years ago.
The key to this is a mandatory security/integrity model. At a given privilege level, you can only run programs trusted at that privilege level. So, if you're running as some kind of administrator, you can only run trusted administrator tools. You can't use a text editor on the password file, for example.
Then you have compartments, and some tools are accessible only in some compartments. For example, the person or program that makes backups needs the ability to read almost everything, but to write almost nothing. (Restoring from backups, which is done less often, requires different privileges.) The security officer can add and delete users, but can't install programs. All this is enforced by the OS, looking at privileges associated with files, users, and programs, not by the applications themselves. A few applications are trusted, and they have to go through an elaborate approval process, which means they're usually rather dumb apps.
The "control panels" used by hosting services are a step in this direction. Users can do some things, and first-line tech support people can do others.
Currently, the big hole is program installation. Installers typically demand far more privileges than they should. In a mandatory security model, installation of an ordinary "application" should mean that the installer has write permission for the vendor's compartment and nothing else.
Smack * (Score:4, Insightful)
Peer Review, Change Control, Auditing, Maintenance Windows, Testing all changes in a lab before production, source and version control / maintenance. These are all best practices, work regardless of operating system and don't require any special software.
Why o why do you want to use software to take the place of established best practices? Best practices are there for good reasons, and those reasons usually have multi-million dollar lessons attached to them. You don't need special software, just a heavy that says yes you /must/ do it this way and raises hell when you try otherwise...
sudo, Xwindows, and logging (Score:2)
There's a relatively simple way to handle this. First, you set up a call center with operators on the phone who each have access to the servers under management. They each have sudo rights to a single command: to create an xterm as root. Their workstations are locked down and do not have an X server installed. (You can take this further and restrict X from reaching them via firewall policies.)
Second, the admins who need root access do have an X server installed. When they need root access to a system, they
Solaris RBAC (Score:2)
No doubt many sites are "lazy", or "old-fashioned" Solaris has had "Role Based Access Control" for many years. Different tasks can be farmed out/delegated to different people.
Auditing, etc. all provided.
I claim prior art (Score:2)
You have to trust someONE. (Score:2)
Yes, you do, even if that someONE is YOU.
For all you quivering sysadmins out there, and you lusers that question their authority and trustworthiness:
Do you trust your CIO to not shackle you to poor choices just for their kickbacks?
Do you trust the Board to not limit your opportunities by failing to act on corporate goals?
Do you trust your CEO to not collude with your accountants and cook the books?
Trust.
Simple (Score:2)
Are there more sweeping yet practical solutions out there for avoiding the weakness of a singular empowered superuser?
Give the responsibility back to the users.
By removing the responsibility from users, one keeps them oblivious to the infrastructure problems. That perpetuates arrogance and the "not my responsibility" mentality.
By moving the responsibility to admins, to the people who do not use (for their primary purpose) the services and infrastructure they are responsible for, make them oblivious to the actual user needs. And self servingly often make the infrastructure more complex than it really needs to be - fe
change control (Score:2)
Use the wikipedia-approach:
Don't restrict, but instead log exactly who does what change when and why, and make it trivial to undo any change.
For example, for /etc use revision-control, and require that all changes be comitted.
That way, yes, the one doing something may screw up, but you can easily undo it. And when the customer calls and goes "why doesn't that work, it worked last thursday!" you can trivially get a list of all changes since then.
As an added bonus looking at logs and commit-diffs give a new a
Re: (Score:2, Insightful)
It isn't about respect, necessarily. I am a sysadmin that has the keys to a lot of things and I have wondered about this very problem. It isn't about how much respect I deserve but it would be nice to a have a distributed method in the event of some sort of catastrophe or something as simple as being sick.
Re: (Score:2)
I keep a log of usernames and passwords for everything in a file in a shared area accessible only by Administrators. A couple of directors have access to it, and one other IT guy.
Re: (Score:3, Insightful)
You keep your passwords in a network share? Are you schizophrenic or just incompetent?
I hope that file is fucking well encrypted ... but even in that case, it's just a bad idea.
Re: (Score:3)
I typically keep that kind of information written down and sealed inside a plain white envelope labelled "Plain White Envelope" in my handwriting and placed in a secure location. If anything happens and someone needs access all they need to do is open that up and use the login information they find inside.
If the envelope is ever opened and I still work there then I need to do a security audit and change all of the passwords. If I don't work there any more then either I have been hit by a bus, or my manag
Re: (Score:2)
Agreed, catastrophes are a huge issue. An additional one I know of is some people with no ability to reciprocate respect, who could one day (or already have) administrative access to systems.
These people need checks and balances. If they decide to quit, they would likely do something slightly harmful for "fun" unless someone got to their account first. Most likely the only safe way to let them go would be to literally "terminate" them.
Re: (Score:2)
It should not be that if you don't, then you lose control of your network.
Re: (Score:3)
Yes. Give a team of admins root access, *SNIP!* have admins use su or sudo to achieve root access.
Given that its trivial to implement, saves a LOT of hassle with shared passwords/keys, using su/sudo should be the default case rather than the special needs case.
Re:sternobread (Score:5, Interesting)
That is how all of our servers are setup. I'm just a "developer" that uses them but I believe no one knows the root password for our systems. It is a *big* random string that is printed out by the sysadmin that sets up the machine, sealed in an envelope with that person's signature on both sides and stuck in a safe. In the event that a machine is so hosed that the root password is needed it is used and then a new one is generated and sealed away again.
Everyone uses sudo for everything. All sudo access is logged.
The system isn't perfect of course, nothing is, but it goes a long way to the worry of one person having root keys for things.
Re: (Score:2)
Re: (Score:2)
Syslog => remote machine.
Done.
Re:sternobread (Score:5, Informative)
sudo logs are almost useless for system audit. Run sudo su - and have at it. There are no logs to follow what actions you perform. Go ahead and craft a sudoers file that eliminates all the ways to load up a shell. Have fun with that...
Re: (Score:2)
So have one of your standard security checks looking for people who open a sudo session to an unlogged shell. If they need a full shell, force them to use a logging shell (ksh-93 with SHOPT_AUDIT enabled can be configured to send to your remote syslog system as one option).
Your security review team needs to be examining sudo logs regularly (daily if PCI). Look for people who abuse their access. Look for what people are doing. This goes doubly true for temporary or limited escalated privilege.
Re: (Score:3)
Unless you are very careful with what commands an admin can run with sudo, there are many ways for him to run a command without it appearing in the sudo log:
sudo vi /etc/hosts
:sh
Now I'm in a root shell and sudo doesn't know anything about it.
Re:sternobread (Score:4, Insightful)
I think you're missing the point. Auditing/logging systems are not meant to provide effective defense. They are meant to let PHB's mark appropriate check boxes on compliance forms and sleep better without worrying what those evil nasty sysadmins are doing. Don't confuse them.
Re: (Score:3)
At work, I worry about these and other issues on a very large scale.
The logging shell must be non-blocking. Will it capture everything? No. If it logs every keystroke, it fails. If it logs everything sent to your screen, it fails. The reason why is left as an exercise for the reader.
The audit requirements have been satisfied for our auditors with command line history logging done in a reliable manner (reliable as in the data once captured is not alterable by the user).
As for X11 only configs, I tell th
Re:sternobread (Score:5, Insightful)
The solution here is to follow a reasonable security protocol in writing the sudoers file. Specifically, the default action is to prohibit. Permitted actions are then whitelisted. On a high-security system, no entry should allow a user to sudo su -. Problem solved.
Incidentally, I see no point in locking down users who have physical access to the DC.
Re: (Score:3)
Why do you think that /bin/bash would be whitelisted?
That said, getting this kind of security is fairly tough because you have to ensure that any utilities can't escape to shell or open files that would in turn allow circumvention. For example, if vim is whitelisted, you can :shell. That can be disabled as a compile-time option. But :r /usr/local/etc/sudoers will allow the person running vim as root to modify sudoers. I don't recall if :r can be disabled, because it's mostly irrelevant--you can modify t
Re: (Score:2)
Well, there's /root/.bash_history
But if your sudo activity log has you doing "su -", then whatever gets borked up after that is automagically your fault as a matter of policy ^_^
Re:sternobread (Score:4, Funny)
Well, there's /root/.bash_history
But if your sudo activity log has you doing "su -", then whatever gets borked up after that is automagically your fault as a matter of policy ^_^
Yeah, nobody's ever altered that file. Also, make sure you are watching for changes to your syslogd config, lest someone disable forwarding, do something snarky, turn it back on. But then, security is rarely something that can be solved definitively by means of one slashdot comment.
Re: (Score:3)
With a team of administrators, you'll have no way of learning for certain who has done what. As you said sudo su - is only one of the many trivial ways. Discretionary access controls as you have described are no better than trusting your admins with the real root password and telling them if you abuse the power you will be fired. At that point, why bother? It's just gonna eat up budget to implement and you are still stuck with the same problem which is accountability. That is to say, who has done what, whe
Re:sternobread (NO security is perfect) (Score:2)
Nothing is perfect but what you described is pretty close. I used to work for an ISP in the 90's and we had something similar. We figured if things ever got "so hosed" we would be booting into single user mode from the console so root's encrypted password was disabled (started with an *) so there was no logging in as root. The closest thing to root was accounts that had the ability to sudo visudo but all sudo was logged remotely via syslog.
This does point out the one security risk that has to be accepted
Re: (Score:2)
another idea for this, though it involves a bit of work per admin - it's nice to keep a separate login and sudo-to-root password.
Create the normal account (i.e. bio) and the admin account (bio-admin) for a given user. The normal user (bio) can only sudo into that user's admin account (bio-admin), and the admin account has sudo access to root. Set up a couple of shell scripts. Move 'sudo' to 'sudo_base'. Create a sudo script which is something like
#!/bin/sh
sudo_base -u "${USER}-
Re: (Score:2)
You're not the only one. I saw "Disemboweling" too somehow though I didn't get Cingular out of it.
Re:why? (Score:5, Insightful)
Not really. It's fun to think I could do anything I wanted, but I don't want to. I like my job, I like the people I work with, I don't want to screw them over. It's nice to have an employer that trusts you too. If I wasn't trusted, I would probably just leave. If they want me to be able to administer and troubleshoot everything, I obviously need full access.
Re:why? (Score:4, Insightful)
if you can't trust the person at the top: then either they don't deserve to be there, or you need to find a new job.
when you're the person at the top: you better have earned the trust and respect of those under you. Subverting it does nobody any good in any long term.
Re:why? (Score:4, Insightful)
Obviously there are those with different goals and standards and it's not always easy to identify them. I'm not sure how to prevent that -- someone who over the years gradually gets more access and one day they decide to go rogue and do something harmful. Even minimizing the attack surface you usually have that single admin account that owns everything else. Maybe I should read the article.
Re:Yes (Score:5, Interesting)
A subset of administrative applications requiring multiple administrators may not be such a bad compromise.
ex:
* change root password (or password to any "wheel" account) - requires multiple administrators to enter the same passwords
*su/sudo'ing to a "wheel" account, or changing said account's privileges, requires the authorization of at least one other wheel'ed user.
* Alterning an active network interface, shutting down, and restarting requires authorization by other administrative users.
Stuff like that, which are things that shouldn't be done often, anyway, and could allow one admin to take over the whole system, seem like good candidates for multiple-approvals. Everything else could be left alone.
The approval process is basically - the root users needs to take the action, and then 2+ non-root (but wheel) users must approve it.
I'm using 'wheel' as that is the group in FreeBSD that is typically allowed access to sudo/su. Not sure how other systems typically work.
Re: (Score:3)
The concept is sound, but in practice the first time there's an emergency where something in the subset needs to be done and 2+ admins are required causing even a small delay, the PHBs will toss it out the window (and not be entirely wrong in doing so). There's always a trade off for greater security/accountability, and IMHO this will cross the line of what's acceptable to management often enough that it won't happen broadly.
Re: (Score:3)
You nailed it on the head.
Companies like accountability. If the thing blew up, they want someone to fire.
with 10+ admins, you cant point a finger in a heat of passion and say "Escort him from the building!", it would take weeks to figure out what happened, and if the 10+ admins were wise they would cover each others asses.
Re:Yes (Score:5, Insightful)
Re: (Score:2, Insightful)
Trust, but verify. I believe the submitter is asking how to provide for verification without breaking operations.
Just as I'd be an idiot for handing my checkbook over to the sole control of an employee based solely on trust, I'd be an idiot for handing over the keys to my IT systems.
Re:Yes (Score:4, Insightful)
Re: (Score:3)
I think there are gaps in management knowledge for most small companies, so they outsource it. Basic accounting is near universal, but tax, for example, is typically outsourced for small companies. Tax prep, however, is via an accredited institution most of the time.
So for IT, do we turn to accreditation of outside providers? Or do we wait a couple generations until basic knowledge of IT is assumed necessary for non-CIO CxOs?
Re: (Score:3)
I disagree. You can instead trust some /people/ with proper checks and balances. This can, in some situations, reduce the risk (for example, if more than one is required for approval of certain things)
Re: (Score:2)
Some times it's a matter of you really can't/shouldn't trust ANYONE but you have no choice. In these cases, a validating/verifying approach can be helpful.
Re: (Score:3)
That is precisely the point of the original question.
We trust politicians with our governance, and over and over and over again, they violate that trust.
Collaborative governance [metagovernment.org] is a way to remove the need for politicians. But it is pointless if we just shift the trust over to sysadmins. They are just as susceptible to corruption as politicians.
We are looking for a way to remove the need for trust in governance: of governments (and any other kind of administration) and of the systems that run them.
Re:Answer in kdawson's tagline (Score:4, Interesting)
"trust but verify"
To get some transparency / accountability, just set up an authlog black hole that includes all of the sudo activity from your servers.
Re:Too many cooks... (Score:4, Funny)
fine, no soup. just type sudo make me a sandwich
Re: (Score:2)
$ sudo make sandwitch
sandwich: target not found
Superuser (Score:5, Funny)
No. Now just hang on a second while I delete your user account and all your data, you presumptuous bitch.
Re:Too many cooks... (Score:5, Interesting)
The submission seems to presume that the system in question is some sort of *nix or Windows box. If we look into the world of mainframe operating systems, we'll see that this has already been fully adressed, and any number of individuals with discrete UIDs may have superuser access. This has evolved out of a history where sysadmins worked shifts, so sharing a single privileged UID/password was/is a bad idea.
The way such access is administrated needs a proper policy within the organisation, though. Back in the '90s, I worked at one outfit (an insurance company) where the vice-CEO demanded superuser privileges despite having no knowledge of system administration or any other computing background. He just wanted to act as overlord as to what staff had access to on their signons. I was very tempted to tell him to get fucked, phrased in more professional terms. Like "Go get professionally fucked".
My immediate boss was (wisely) more inclined to a diplomatic approach, however, so he pursuaded me to install a dummy program for him that was enough to convince him that he had what he wanted, without granting him any kind of command line access, or ability to change system configuration.
Different Objectives for Dividing Responsibility (Score:3)
There are a number of different objectives people might have for splitting up superuser powers, and depending on what you're trying to accomplish, there are different kinds of solutions out there. For instance
Re: (Score:3)
A very talented, and very honest person will not put up with layers of approvals and constant monitoring.
Have you actually worked at a company composed of very talented, very honest people who put up with this very thing every day? Setting up an ITIL-compliant change management system -- and getting everybody on-board with using it -- is a very daunting procedure. Speaking as someone who has been on both ends of this, I can say that in the end, it's worth it. My day as a sysadmin is no longer all about putting out fire after fire, dashing around and pulling crazy hours at the whims of vice-presidents who th