Remote, Automated Configuration of Unix Boxen?

drift factor asks: "I work for a small company with about 200 Linux desktops, all identical aside from hostname and IP information running Redhat from a kickstart install. Currently, when something needs to be changed on all of them (say, the default gateway) we either have to ssh into all of them and do it manually, or write a Perl script to do so for us. Is there a good remote configuration system that will also allow some level of automation, i.e., I tell it what IP I want the default gateway to be on each machine and it updates them for me?" If you manage a large server farm and have used or written utilities that do this, please share your experiences.

This could work

[Anonymous Coward]

This is actually a good idea, the trick is to run your own Debian mirror (at least a partial mirror). Add it to /etc/apt/sources.list (on each computer), and make some virtual packages: standard-server, standard-workstation, etc. for example. When you want to install something on all computers, add that package as a dependency. You can just update the packages to modify stuff.

"apt-get update ; apt-get dist-upgrade" would need to be done every day, but try to get each machine doing it at a different time so you don't overload your server (or network). You could either use random times, or something like anacron so you can say "once a day" instead of "2:00AM every day". Doing it at night would be good if you leave the machines on.

Pretty easy

[Anonymous Coward]

It's done all the time. Just do what my boss did: hire a flunky to take care of it. Then, you just bark out orders: "upgrade Netscape", "change the default gateway blah blah blah", "get me a cup of coffee", etc.

Re:DHCP is obvious

[Anonymous Coward]

How about modifying the startup scripts so each computer contacts the server on bootup, and runs a script from an AFS-mounted partition. Just modify the script depending on what you want to do (you'd need to detect if the script already ran or not, something like "touch /etc/updates/netscape-bo-fixed").

This is basically the same as your idea, except the client downloads the script from the server (instead of the server sending stuff to the client).

A lot of stuff

[Matthew Weigel]

For network configuration, DHCP is one reasonable alternative (although you'll probably simply want bootp support, allowing systems to keep their own IPs static). Using rdist to propagate changes may also be reasonable, for other things, and it can be used with ssh.

Kerberos is one way to handle network-wide logins, although I have issue with it because it requires /etc/passwd be kept around (Kerberos handles authentication but not the additional needs of Unix, such as file ownership determination and such). NIS is insecure, NIS+ is too complex for a single domain (and I'm not sold on its scalability, either). However, it doesn't look like that sort of thing is really you're concern, so I'd say rdist :)

Re:DHCP is obvious

[meldroc]

For something like this, you may be stuck writing a script. The reason I recommended Expect is because this tool makes these kinds of scripts incredibly easy to write (especially compared to writing the same scripts in Perl). Give it a shot.

I'd also suggest finding an RPM of the updated netscape, then the code for updating Netscape in Expect would be:

spawn "rlogin $hostname\r"
expect "#" ;#root shell prompt...
send "rpm -Uvh netscape.rpm\r" ;#RPM switches right?
expect "#"
send "logout\r"

Missing the point

[BigWillieStyle]

I think a lot of folks here are either missing this guys point, or haven't ever had to deal with this situation. Here's an example (personally, I'd probably point to cfengine (I'm about to need similar capabilities)).

Say I've got a (solaris) box named "foo", the /etc/system file on "foo" looks like this:

set semsys:seminfo_semmni=10
set semsys:seminfo_semmnu=300

and then I've got a (solaris) box name "bar",
the /etc/system file on "bar" looks like this:

set semsys:seminfo_semmni=30
set semsys:seminfo_semmnu=700

Now, let's say that I want to add the following lines to both boxes:
set sd:sd_io_time=0x3c
set sd:sd_max_throttle=2

I don't want to copy over the file, I like the values how they are! That's the basis of this guys question, and the same thing that I'm looking to do quite soon.

None of the options using CVS or DHCP take this into account. Please make sure you understand the full nature of the post before you add 50 lines of worthless advice.

Re:Yes

[kableh]

This [securityportal.com] article I read the other day on Security Portal was a good read. It was mostly about using ssh but gives some examples of how to use it to automate some sys admin work and do it all securely as well.

Re:Yes

[drift factor]

Ok, so giving the default gateway as an example was a bad idea, but you miss the bigger picture: I need to be able to change a lot more than just network configurations on all the machines. This could a pam configuration file, installing a new package, virtually anything. So far, the thing that looks the best to me is cfengine [hioslo.no].

Re:DHCP is obvious

[drift factor]

NIS and NFS are already implemented, and we are migrating to AFS and kerberos. Keeping user accounts in synch and having a unified directory structure isn't my problem. Consider this scenario: The Brown Orifice HTTPD bug is discovered in Netscape, which all of my users use. I obviously want to fix this so that no users are vulnerable. NIS helps me not. Short of using a script to login to all the boxes and perform actions to update netscape/disable java, what is the best way to do this?

DHCP is obvious

[bluGill]

For your network dhcp is the obvious solution. (Congradulations to the first post comment that was on topic an correct. lacking details, but that is a different story)

For a more general approach, NIS comes to mind, though it wasn't designed to be secure. NIS+? kerboses? Your not the only one with the problem you state, in fact you hardly have a program compared to many installations. I've seen systems with 300 Solaris/aparc machines, 200 sunOS 4.x (much nicer then solaris IMHO), 75 IRIX, 200 Linux, and probably some HPUX and AIX scattered in there two. This at a university for student accessable accounts in CS or other engineering areas. Your 200 machines is nothing compared to the 20,000 users they managed. And most would not consider that close to large.

My point is that others have seen your same problem, and worked on solutions.

Almost the same idea

[tolldog]

I use SystemImager [slashdot.org] to manage all of my machines.
I build the image on a box and the propogate the changes via a rsh/perl script that tells the clients to update.

For large system updates and configuration, it isn't that bad of way to go.

Depends on the complexity of the task

[Tet]

For changing a nameserver, for example, you could just rdist or rsync /etc/resolv.conf to all the machines. However, it gets more complex when you want to modify a file rather than overwrite it. To change the gateway, as in your example, you'll want to modify the GATEWAY parameter in /etc/sysconfig/network, while leaving the HOSTNAME parameter untouched. The only practical way to do this is to write a script, but then you're doing that already. Writing scripts is The Unix Way (tm) :-)

Warped thought - cron and CVS.

[Christopher Thomas]

Here's a warped thought - set up a CVS server with all config files and applications that you care about updating checked in.

Set up a cron job on each machine to check out the latest version of the installation every day at 5am. Make the cron job shut down and reinitialize anything important, too (or just have it reboot the machine and let init/shutdown scripts take care of it).

This isn't a remote admin solution, but it _does_ let you easily make sure that all packages and config files on the machines are in synch. Upgrade Netscape on the master server, for instance, and the other installations migrate over in a few hours.

You could even rig it so that the cron job calls a specific script before completion, and check that script out of CVS for any specific shell-ish things that have to be done for maintenance to complete the update (test these scripts carefully, though).

cfengine works pretty good for me.

[Sir Spank-o-tron]

We use cfengine and rsync.
we have a master server that get's config files from CVS, and pushes out to rsync servers.

every host calls rsyncs once an hour (or as often as you can get cron to do it) from the rsync servers, and EVERY hosts config is in cfengine, even the rsync servers and cvs servers.

we make heavy use of CNAMEs so that if a machine disappears, we can magically recreate it and none of the workhorse hosts know the difference.

go check out infrastructures.org for some good reading.

Expect can be your best friend in automation.

[meldroc]

I work in a testing lab where we have to run large numbers of regression tests that would be incredibly time consuming if we didn't use automation tools.

I've been using Expect [nist.gov] for test automation on the latest series of tests. Expect is an automation scripting tool that runs on top of Tcl. It is very useful for automating command-line based applications using "send &ltstring> expect &ltstring&gt" sequences.

As far as licensing is concerned, Expect was developed at the NIST, and has been released into the public domain. That's about as free as you can get.

Re:Yes

[Zurk]

i use scp to copy the rpm files over, use ssh to login to the box with a script and install files by running rpm -Uvh etc etc by piping the stuff in thru the script. its pretty secure that way and works nicely. also easy to have a set of commands for doing various simple things and letting the script pipe those with a command option. i.e. script install whatever.rpm installs the whatever.rpm file, script cp file1 file2 copies the file etc etc.
I also use procmail to do some grunt work (weird huh?)...for example if i want to clean the print q of server 26 i can mail root@server26 with the header as COMMAND: DELETE PRINTQ and a procmail script will see the COMMAND: in the header and do a simple lprm -Plp -...ok..im half asleep as i type this..but i hope im making some sense. im a uni admin BTW. i've got machine with weird configs to look after so it might not suit your problem...none of my machines are the same.

cfengine!!!!

[The Madpostal Worker]

cfengine is meant for this sort of thing, with it you can easily mirror config files accros the network, have it tidy files, and do trip wire like things. Its a great product, and its free (in both senses). Basically, you tell it how to do tasks in a class sense, this is how you upgrade perl on a solaris box, this is how you do it on a linux box. As a result, you can share code between classes(on a solaris or linux box, do this). Its a *great* system.

/*
*Not a Sermon, Just a Thought
*/

Powerful tools...

[Ryan Kirkpatrick]

There are two tools that I know of that might be of help. They are:

• tut: "Tell Unix To..." by Jim Barbour. A command that will run a non-interactive shell command on a user-defined class of machines. (No URL handy, sorry).
• cfengine: A generalized, powerful, shell like language for dealing with large numbers of machines. (http://www.iu.hioslo.no/cfengine/ [hioslo.no])

I have never used either, but have read some documentation on cfengine. cfengine looks like it would be more of an automative tool than tut, while tut might be handy for quick and dirty updates. Anyway, my two cents.
------------------------------------------ ----------------------------------