Keeping Audit Trail of Activities from Root Login?

supersam asks: "This question might sound a bit naive, but I am comparatively new at this. So, here goes: on one of my application, based on UNIX platform, I need to implement a mechanism by which, I can trace when an access to ROOT was attempted and what all actions were performed during that session. While info on the first part is probably available through UNIX logs, the second bit is difficult to me. The need is arising because, while it is useful for the support team to know the ROOT password as most of the wonderful things can be done with that access, it also can be a nightmare if something goes wrong and we have to trace the culprit or the root cause of the problem (config files etc...). Is there any way of keeping an audit trail of all activities performed from ROOT login in UNIX which can be traced." Interesting thought. About the easiest way to do this would be to give root it's own custom shell and log everything that shell does. Are there shells that can already do this? Has anyone else implemented such a system? If so, what did you do?

Sudo

[ffsnjb]

Force people who want/need to have root access to use 'sudo'. It will log all activities using syslog. This also negates the need to give lowlevel admins the root password (yay!).

Sudo is in the ports collection for FreeBSD, no idea about linux though.

Re:Sudo

[ryants]

Sudo is in the ports collection for FreeBSD, no idea about linux though

There are RPMs or DEBs for sudo... the package name is (originally enough) "sudo".

sudo pacakges [rpmfind.net] at RPM Find.

Re:Sudo

[NNKK]

or, thanks to a freshmeat search, you can go to sudo's homepage and get tarballs
http://www.courtesan.com/sudo/

Re:Sudo

[staplin]

Though you still have to be careful with a few things...

like "sudo sh" or "sudo su" where the only log you get is that a particular person ran sh/su, but not what they did after that point.

But IIRC, you can configure sudo access to specific programs/commands, and can block access to ones that cause problems like this.

And as long as you send your syslogs to a remote/secure machine you don't have to worry about a sudoer editting the logs to cover up after themselves either.

Re:Sudo

[Saint Nobody]

not only that, but there's a million other programs out there that allow for access to the shell. less, for instance. it could look like somebody is innocently reading a config file as root (well, innocent as long as that's part of their job...) but they could use shell escapes to do just about anything. before you give anybody access to anything using sudo, think about how it could be used to obtain a root shell.

and i'm not talking out of my ass here. i've run into situations at work a couple times when i couldn't find the full time admin and needed to get something done, so i used my sudo access to get a root shell. (of course i had been given permission to root our boxen in case of emergency)

Re:Sudo

[randombit]

and i'm not talking out of my ass here. i've run into situations at work a couple times when i couldn't find the full time admin and needed to get something done, so i used my sudo access to get a root shell. (of course i had been given permission to root our boxen in case of emergency)

Yeah, that's the problem with sudo. If you configure it so anyone can do any measure of useful work, then they can also crack root with it.

Of course that's just re-affirming the "golden rule of Unix", don't give root access of any sort unless someone needs it and can be trusted with it.

We use it here mostly as a nice convinience, so we don't have to su back and forth (we also set it up so we can use rpm through sudo without a password - again convinient, but not secure).

Policy solution

[William Aoki]

Since this is access for (supposedly) trusted techs, you could simply let them know that they had better have a very good excuse if you catch them invoking a shell and running privlidged commands without logging. Where I work, members of %wheel have an agreement that we use sudo but not 'sudo -s' (unless it's something like 'sudo -u test -s') or 'su', so that an audit log is kept of what commands have been run. More than once the sudo log has been useful for problem hunting.

If you don't trust your techs or you need to give out limited elevated privlidges to untrusted users, then none of what I've said applies.

Re:Sudo

[FortKnox]

even worse. "sudo vi", then drop to shell in vi (works with emacs, too). Too many workarounds with sudo.

Re:Sudo

[Hardware]

Or "sudo vi" or "sudo less" or anything else that lets you invoke a shell.

Re:Sudo

[sql*kitten]

But IIRC, you can configure sudo access to specific programs/commands, and can block access to ones that cause problems like this.

That's fiendishly difficult to do. Let me give you an example, what could be more innocent than the "cat" command, so a user can read the syslog for debugging? Well, "sudo cat" can be used to view or rewrite /etc/shadow, that's what!

The problem really is deeper, in the Unix "philosophy" itself. You're either uid 0 who can do anything, or a regular user. There's no notion of really fine-grained access control. Wouldn't it be great, for example, to be able to grant a group of users the right to bind to port 80 (for example) without having to be root in the meantime? Or to create a group of users who can start and stop sendmail/popper without being able to read mail spools? There's simply no elegant way to do this - the users/groups r/w/x mechanism on the file systems just isn't flexible enough. VMS solved this problem decades ago, but its access control lists (on file system objects and system privileges) never cross-pollinated into Unix, similarly processor quotas and many tuning parameters.

The only way to log what root does is to use a packet sniffer on a box that the root user does not have a login on, and physically secure this machine, and the main servers so they cannot have their keyboards accessed. In this case, even the use of SSH would have to be a "warning flag" that something suspicious was happening. You can't even rely on a kernel mod to track root, since the kernel can easily be replaced.

Re:Sudo

[extra88]

Ditto, sudo is your friend. There are a couple of things to be aware of. First, you can tell sudo to log to a separate file if that's what you want. Second, sudo has a -s switch which gives the user a shell for root. If they use that you won't have a log of what was done in the shell. I believe the use of -s can be disabled.

There's also the good ol' .history file, you could do something archive the .history file for root.

Re:Sudo

[loncarevic]

1. sudo for linux exists for centuries

2. how you will record when they type "sudo su" ?

3. use process accouting programs, every distro have it

i

Related to "Handing Over Root Passwords?"

[MrBlack]

This isn't related to the "Handing Over Root Passwords to Consultants" ask slashdot a couple of items below is it? If not there may be a few good suggestions there.

Try process accounting

[Anonymous Coward]

For the second part of your question, you could enable process accounting. Which logs accounting information (including the UID) for every process that successfully terminates. See the man pages for acct(2) acct(5), accton(8), and sa(8).

Re:Try process accounting

[Bryan Andersen]

Enabling process accounting only gets you what commands were run. You don't get the arguments or actions taken within those commands.

Re:Try process accounting

[fsck!]

That is trivialy easy to circumvent. Process accounting only kicks in when a process exists. So as long as /tmp/.. /sploit exec()s ls instead of quitting, it will never reach accounting. Unless you notice that something claiming to be ls has taken 10 minutes of cpu time in under a week, or something like that...

Sudoer

[LABTYD]

Linux has Sudoer which you can give root to a user and they use theey're password not root
and you can actually set how much root access
you want them to have

Hard Copy

[cehardin]

If the system is not accessed physically then you can set up the logs to print out to an old-fashioned dot-matrix printer, even root access can't erase those :-)

Re:Hard Copy

[larien]

Until the time he putzes about in root and causes the printer to run out of paper :)

However, it is a valid method of logging to have syslog print to a printer; it means that if a cracker breaks into your system, you still have a log even if he deletes/modifies the log files.

Gives me an idea....

[invenustus]

However, it is a valid method of logging to have syslog print to a printer; it means that if a cracker breaks into your system, you still have a log even if he deletes/modifies the log files.

It seems like there should be another way of doing that without killing a million trees a day. Maybe you could have a dedicated logging box. Every time another box makes a syslog call it could send it out as a little HTTP POST to the logging box. Of course, once the logging box gets cracked, you're no better off than you were before. Maybe there's money to be made in a "log farm" that writes all posted data to a medium that can't be rewritten immediately even if the machines get rooted - tape might be a good choice. Or make a device that punches holes in paper but in a compressed readable-only-by-machine format that saves trees.

I'll add these to my list of Things I'd Do If I Had The Time And Weren't A Lazy Fsck.

Re:Gives me an idea....

[Cryptnotic]

Actually, you can set up the remote logging box to only accept log connections and deny all others (especially remote logins like telnet, rsh, ssh, etc).

Cryptnotic

Re:Gives me an idea....

[Lancer]

man syslogd

On a less sarcastic note, remote logging is a very good idea. Security Focus [securityfocus.com] has a new mailing list about Log Analysis [securityfocus.com] that would probably be of some interest to you, as there's been quite a lot of discussion already about remote logging.

You're right about one thing - it does save lots of trees :)

Re:Gives me an idea....

[hey_robot]

Why not just have the machine "print" out the log to the parallel/serial and have the other end connected to a machine that just reads what is sent over the parallel/serial and logs it? Make this machine only accessible directly and not over a network/modem/etc. Just make sure you have enough drive space! ;) -- Dusty

Re:Gives me an idea....

[invenustus]

Why not just have the machine "print" out the log to the parallel/serial and have the other end connected to a machine that just reads what is sent over the parallel/serial and logs it?

That's an excellent idea, but I'd still be afraid that if the main server got rooted, you could do some kind of DOS to the log box just by sending it too much crap. I think to really do this in a foolproof way, I'd want to keep the logs on a medium that could ONLY be destroyed or erased by accessing it in meatspace....

Sudo Homepage

[staplin]

Information about sudo, including docs, downloads, and mailing lists, is available at http://www.courtesan.com/sudo/index.html [courtesan.com].

Snoopy

[Spacelord]

There is a "wrapper" library called snoopy that can do this. It logs all commands executed to syslog. You can then let syslog log over the network to a dedicated logging host, to which your users don't have access.

You can find snoopy here:
http://www.citi.umich.edu/u/marius/snoopy/

Use a secure unix

[SuiteSisterMary]

One of the requirements of the higher ratings of the Orange Book is copious logging. Ridiculous amounts of logging.

shell

[N3MCB]

I'd be carefull with putting the real root into a wrappered shell - too much can go wrong early in the boot process and a wrappered root account might not be functional depending on what pieces parts are borken. I'd create a seperate account for these people and then wrapper that one. The sudu spproach still sounds better to me as more secure in the long run but I don't think the logging was geared towards a malicious root user who would delete the logs, rather it was directed at an incompetent one that would screw the system up then hide to avoid the resulting chaos. In the latter case the wrappered shell would be less work to set up/maintain.

Some fairly simple ways.

[AtariDatacenter]

These are some halfway decent ways of tracking what is going on with the root shell. Mind you, they aren't hackproof, but they make decent information for after-the-fact investigation:

Of course, enable your shell history. Also, turn on "lastcomm" so you can at least get the commands (but not the arguments) given at the shell prompt. Finally, more advanced versions of UNIX (I personally know HP/UX and Solaris do this) allow you to turn on snooping to system calls so you can log to disk all the nice commands and their arguments, like fork, accept, all types of disk access, etc.

Solaris has this included in something called "bsm" (basic security module) which comes with the OS. HP/UX you have to put into "trusted" mode and enable it through SAM. The logs it creates are very difficult to make an authentic looking fake for. Possible, though. Yes, syslogging to a remote host is a good idea.

Of course, a nice hacker would simply flood the syslogd that is keeping the logs... but you've got to decide how far you're going to go with all of this.

Also, there are some tools, like Sun's Configuration Service Tracker, which regularly scans the system and logs configuration changes.

Maybe there is a better way of doing all of this. But the layered approach seems to catch a lot of different things at a lot of different levels. All depending on how far you want to dig down into things.

Modify Bash itself.

[ASyndicate]

/etc/shells contains /bin/bash

I help (in spare time) Administer a shell provider (Techcellent.com [techcellent.com])

What the admin did was modify bash itself.. Making a small portion of the code log something to a very obscure place that cant be accessed by anyone. Try that!.. But I think the line printer Idea is a Very Good Idea and requires the least effort

Re:Well uhm..

[LYNX99]

In addition to /.bash_profile, don't forget about the HASH Table (Command Logging) as well

Take the log out the computer

[madumas]

To prevent somebody deleting the logs, you have to put the log entries out the computer.

There are many ways to do so:

• Email every line (with a good filter on your email client or procmail it's easy to manage)
  
• Log on an other server (not with nfs, but anything with "append only" access... a cgi script on the other would work well)
  
• Printer (take physical place, not easy when colocating)
  
• Pager (At least, receive a page for every root login or shell process ran by root)
  

Using something like "tail -f nameoflog.log | scriptsend.pl" would work well.

---

Re:Take the log out the computer

[J'raxis]

tail -f nameoflog.log | scriptsend.pl

Be careful with this, if the script is set up naïvely (using <> to grab all input and then send in one command) the script would keep buffering until `tail` exits.

You could have `tail -f` write to a named pipe (FIFO), and a script run every minute or so (crond), reading from the FIFO, and send the mail. `tail -f` would keep writing to the FIFO. Each time `scriptsend.pl` tries to read from the FIFO, it would get all the data, get EOF properly (FIFO gets cleared), send, all the while `tail` keeps writing to the FIFO. On the next run, the FIFO has new data in it for the script to use.

% mkfifo /var/log/fifo
% tail -f nameoflog.log > /var/log/fifo

And, every minute:
% scriptsend.pl < /var/log/fifo

Re:Take the log out the computer

[debaere]

Wouldn't it be easier just to tell syslogd to send the logs across the network to another computer? I believe that this is a native function of syslogd

Include 'script'

[trexl]

Why not include the script command in root's bashrc and name the file name the date and time of the login, or whatever other information that you would want in the name? Put it on some special filesystem and then all is good.
Just a thought?

Re:Include 'script'

[opk]

That would be nice because you'd get all the output and hence the context of the commands. The trouble is if you don't want the user to know they are being logged. Script outputs 'Script started, file is output' and to remove that, you'd have the modify and recompile script. You would need to do `exec script' from the .profile not a .bashrc because in the .rc you'd get and infinite loop and without exec, they could quit script and still be in the shell.

I'm hoping to do something like this to see what our DBA (who works directly for the client) is doing on the Sun server which I administer. She keeps changing things without telling us and it is getting annoying. Unfortunately, she knows about .profile so I can't modify that without her seeing the changes. So far, I've just kept an eye on .sh_history.

Modified shell writing all to syslog.

[shippo]

A few years ago I used to often need root on Banyan VINES servers for debugging/maintenance issues.

The Bourne shell used by root was modified to log all commands to a number of files in a particular directory (/etc?). The files were written to in a circular fashion - when one file was full, the next would be overwritten.

The really odd thing was that these logs were not written to immediately, but were flushed out sometime in the future, often after the shell had been closed. I never got to find out how this worked, but I assume that the syslog equivalent was responsible.

Of course on this system there were very few tools, and most of them were derived from mid 1980s code. I'm not even sure if it was possible to get a shell from their implemantion of vi - it was a very broken vi implementation.

This logging was pointless, however. The majority of root access was used to query and modify the various security databases via a number of text based tools. If run interactivly these would not log anything.

Or try another approach ...

[CodeSpotter]

How can I log so that I have a trail left to examine?

This question is easy to answer: turn on logging as much as possible. The only downside is that anyone with root access can alter or delete those logs. Process accounting as a means for logging what root is doing, is easy to implement but very hard to use. PA only logs when a command finishes, so most of the time you will have all the commands in the reverse order they were executed. And it doesn't log arguments. So it certainly does not give you a bullet proof solution. The best solution here is to try logging to a remote computer, syslog has facilities to do just this. Using sudo, a well known tool on Linux, is also an option. Using sudo however is not that easy as it seems. You have to carefully enumerate the list of commands the user can execute. Enlisting "vim" for instance, opens up sudo to every command since in vim you can start a shell as root in that case.

Another interesting point to mention here is that it is probably better to prevent then to cure. Abuse of the root account, a break in, ... can be made very difficult but not impossible. Limiting the damage in such a case is probably the most important step to take. Make sure that whenever such an event occurs, you can go back to a safe and functional server using backups or anything similar.