Version Control for Important System Files?

TokyoCrusaders92 asks: "Like a lot of other organizations (800 staff, 5000 students) we have a mix of Windows, Novell & Linux (primarily Linux) for our IT infrastructure. We now have a multitude of config files, firewall rule bases, shell scripts, and so forth which are managed by multiple people and groups. Recently, we started using RCS for version control of the firewall rule-base, but this doesn't seem like it would scale up to larger groups of users. While thinking about this, it would seem that the critical features would include: version control; logging; multiple users; secure authentication; and integrity checking. What are other people using to manage their config files?"

Notepad

[Timesprout]

What more could you possibly need?

Re:Notepad

[smallfries]

GP uses a humorous post. Technical criticism is not applicable and is thus useless.

Re:

[Timesprout]

My faith that Slashdot is not populated by humourless pedants is restored.

Re:

[fbartho]

Mine wasn't, I forgot to use version control.

Distributed SCM

[MichaelSmith]

I keep my config files in a directory structure in my home directory on my laptop which mirrors the structure of the systems I maintain. I use the mercurial [selenic.com] DSCM for version control and push revisions to a user account on each server. From there I run a script as root which recursively copies the files into the target directory tree.

Re:

[Breakfast Pants]

If you have to handle different files for different machines, I'd reccommend Oracle's opensource cvsman

A CVS server

[kotj.mf]

How similar are your systems? I help manage several thousand distributed boxes that are reasonably identical, and we keep everything in a central CVS server: management scripts, config files, crontabs, what have you. There's no reason it couldn't be used for more heterogeneous systems, other than having to be more careful with file naming conventions.

Use SVN instead

[Schraegstrichpunkt]

For a new installation, Subversion is probably a better choice than CVS, mainly because changesets are committed atomically, directories are versioned, and it has better security when dealing with remote access.

Re:

[Steve Folta]

He'd be talking about things like Git, Mercurial, Darcs, Monotone, Bzr... See http://en.wikipedia.org/wiki/List_of_revision_cont rol_software [wikipedia.org]. VCS's have made great strides in the last couple of years (probably due to the well-publicized needs of the Linux kernel). I'm really surprised that most of the comments here keep talking about obsolete systems like Subversion.

Re:

[tzanger]

I tried CVS for that... it is too much of a pain in the ass to patch CVS to accept root users. I don't know about you, but screwing about with permissions just to appease CVS was not my idea of fun. Maybe next time I'll run svn or something to see if it works better for config file management.

Re:

[jgrahn]

I tried CVS for that... it is too much of a pain in the ass to patch CVS to accept root users.

I manage /etc on all my boxes with CVS, and I see no such problems. The CVS used is various versions, plain and unpatched (except by Debian).

Subversion

[Just Some Guy]

Next question?

Re:

[Threni]

> Next question?

Can we have version control in the file system, please? That way we'd *never* accidentally delete/change files, have them corrupted by viruses/hackers etc. You could label all your files each time you boot up. Perhaps for speed there'd be some files you'd not bother with (swap file, registry, temp files for some apps) but generally it would be pretty cool.

Re:

[maxwell demon]

Can we have version control in the file system, please?

It seems so. [google.de]

Those who don't know VMS...

[MrBoombasticfantasti]

Those who don't know VMS are wont to re-invent it... ;-)

See Files-11 [wikipedia.org] for a flashback.

Re:

[bill_mcgonigle]

Those who don't know VMS are wont to re-invent it... ;-)

ZFS seems to be headed in that direction, though I've seen bickering about how it would be exposed. No ;1, though, please. ;)

Re:

[morgan_greywolf]

Unfortunately, ZFS can only be used on Linux through FUSE. 'Real' kernel module support for ZFS on Linux is not possible due to several Sun patents and the fact that the ZFS code is CDDL, which is incompatible with the GPL.

Re:

[bill_mcgonigle]

Real' kernel module support for ZFS on Linux is not possible due to several Sun patents and the fact that the ZFS code is CDDL, which is incompatible with the GPL.

Fortunately OpenSolaris will be licensed GPL3 soon (at least that's the word on the street).

I'm not sure what happens when you try to use GPL3 code in a GPL2 product like Linux. I'd have to guess the FSF has that covered, but I'm not sure. The patent part should be covered regardless.

Now that I think about it, Patents + GPL3 is a nice way for co

Re:Subversion

[Cyberax]

Subversion is the ideal solution - because it needs a lot of junk in .svn directories :( And it can mess with some scripts that do recursive grep or something similar.

SVK is better, but it is not as widely supported as SVN.

Re:

[Just Some Guy]

Subversion is the ideal solution - because it needs a lot of junk in .svn directories :( And it can mess with some scripts that do recursive grep or something similar.

That's true, but in practice is that such a huge problem? We use SVN already for all project code and have managed to make do with it. If you can grep and operate on a few megs of in-house development source, a comparatively tiny /etc directory shouldn't be too much hassle.

SVK is better, but it is not as widely supported as SVN.

That's a

Re:

[Cyberax]

Grep doesn't have a problem with /etc size, but .svn directories store the pristine copies of controlled files. So you'll get TWO matches instead of one.

SVK is built on top of SVN and written in Perl. It works pretty much everywhere, but you don't get nice GUI clients for it.

Re:

[xlv]

Subversion is the ideal solution - because it needs a lot of junk in .svn directories :( And it can mess with some scripts that do recursive grep or something similar.

That's true, but in practice is that such a huge problem? We use SVN already for all project code and have managed to make do with it. If you can grep and operate on a few megs of in-house development source, a comparatively tiny /etc directory shouldn't be too much hassle.

I've been using svn for a few years now for projects and system files.

Re:

[chris_eineke]

because it needs a lot of junk in .svn directories :(

And CVS doesn't?

And it can mess with some scripts that do recursive grep or something similar.

And CVS doesn't? tar has --exclude=CVS or --exclude=.svn and -- are you surprised yet by the UNIX way? -- so does grep.

Re:

[Cyberax]

Why do you think that I'm protecting CVS?

I don't like --exclude because it leads to more typing and also can lead to errors if you forget to add it.

Re:

[chris_eineke]

can lead to errors if you forget to add it.

But that's an universal truth. If I forget to pull down my pants to take a dump, I will crap my pants. :-)

Re:

[Cyberax]

Yet one more reason not to wear pants [nopantsday.com] :)

And, personally, I prefer to minimize number of 'failure points' in my workflow. I take Murhpy's laws very seriously.

Re:

[ceroklis]

RTFM: that is what the export [red-bean.com] command is for.

Re:

[Cyberax]

I know about 'export' (I've been using Subversion since 0.24 alpha, AFAIR :) ). But usually you need to tweak something directly inside /etc, check if it works and then commit the fix to repository.

Exporting and importing changes to working copy leads to more errors (if you forget to delete a file from repo, etc.).

puppet... maybe (not yet at least)

[getha]

Using something like subversion or any other version control system for such a task just leads to Yet Another Homebrew Administration System, that will probably lead your successors to tears and insanity. Use tools already there, and that are pertinent to the job.

version control; logging; multiple users; secure authentication; and integrity checking.

All those features you need are mostly already there in puppet: http://puppet.reductivelabs.com/ [reductivelabs.com] (and maybe also in cfengine, but that's a nightmare). And the development on puppet is really picking up steam at the moment.

Problem for your situation is that it has no Windows or Novell support as of yet, but recently work on Windows at least seems to have started. And if your first priority is mainly config file management: that part should be fairly trivial.

Our system is great

[Heir Of The Mess]

We have a login script that calls another script that is the one that gets modified. To use different script you just change the name of the called script to the one you want. If the script gets changed then the person who changed it changes the name and we can tell how old it is by the date stamp on the file. So far we have these (oldest to newest):

sublogin.bat
sublogn2.bat
sblg2fix.bat
latestlg.bat
newlatst.bat
finalfix.bat
reverted.bat
fixwrked.bat
NtOnMyPC.bat
WksOnMyn.bat
NTONMYPC.bat
TryThis1.bat

Seriously though, subversion is good because it lets you do atomic checkins.

Case insensitivity?

[mosel-saar-ruwer]

NtOnMyPC.bat
WksOnMyn.bat
NTONMYPC.bat

Do not M$FT file systems tend to be case insensitive?

Re:

[Heir Of The Mess]

The command line and shell are case insensitive, but the names are still saved with the case that you typed in.

Re:

[Woy]

Hence you can't have 2 filenames differing only in capitalization??

Re:

[Heir Of The Mess]

Hence you can't have 2 filenames differing only in capitalization??

Just tested trying to create Test.bat and test.bat in the same directory in Windows XP using the "Explorer" shell, and yes I couldn't create the second file - said the name already existed.

Re:

[jdray]

AFAIK, there's a system-wide switch for later versions of Windows (post W2k?) that enforces case sensitivity. Don't care, though; I use a Mac.

Re:

[Heir Of The Mess]

Yeah you have to add this value to the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Cont rol\Session Manager\kernel

"obcaseinsensitive"=dword:00000000

Which is what you need to do if you are running Services For Unix

Re:

[Anonymous Coward]

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Cont rol\Session Manager\kernel

"obcaseinsensitive"=dword:00000000

What an abortion the Windows registry is! If MS followed the *nix philosophy, the setting would be in an /etc file with a simple setting called

case_sensitive = false

This would of course be fully documented in a clear, informative man page.

Re:

[26199]

How in the world did this get modded interesting?

It should be clear to anyone that you were being sarcastic :p

Add another box....

[williamyf]

Get an "Industry Standard OpenVMS" (that the name HP gives to that beast) and use it as your main file server and config info repository. All the features you desire are there by default.

While this response was accurate from the technical point of view, it was meant to be a joke

rsnapshot

[perlionex]

I use rsnapshot [rsnapshot.org] to do version control of my entire system. From the description:

rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.

Personally, I configure rsnapshot to generate snapshots every 4 hours, and then daily, weekly, and monthly.

In your case, since you only want versioning for your configuration files, you can point rsnapshot at just the configuration directories (probably just /etc).

Re:rsnapshot

[hattmoward]

Those are snapshots, not versions. It is possible for a particular revision of a file to roll off your monthly rotation and be forgotten forever.

One idea for Windows and Unix (shameless plug)

[DNAGuy]

For Windows, the default answer for deploying configuration is Group Policy [microsoft.com]. With the Group Policy Management Console [microsoft.com] you can back up your GPOs and store them in your version control system. Alternatively, you can use a specialized Group Policy version control and troubleshooting product like Group Policy Manager [quest.com] from Quest [quest.com] which allows offline editing and testing before deployment to your production environment.

You can actually extend Group Policy to your *nix and Novell machines as well using Vintela Group Policy (a part of Vintela Authentication Services [quest.com]), which also has features to allow your *nix machines to use your Active Directory accounts. If you use it in combination with Group Policy Manager, you can back up the *nix specific settings alongside the rest of your GPO. Also, the integration allows you to diff different versions of your *nix settings alongside the rest of your GPO.

[Disclaimer: I am a developer on the Group Policy Manager team at Quest. While I'm proud of my product, I'm not any kind of official spokesperson for my employer.]

rcs

[pe1chl]

On systems where it matters, I keep config files etc in RCS.
In each directory where config files live that I want to keep, I create an RCS directory and rcs -i the file(s).
Nightly, I job runs that finds all files for which an RCS entry exists and that are newer than that entry, and a copy is checked in.

No need to think about checking in/out all the time, no problem that the RCS seems to believe that you don't want to keep the actual file around.
It does not save every edit but at least I have a copy of each day's state of the file.

WSUS for system updates

[Joe The Dragon]

will be a big help

Darcs

[swarsron]

Try darcs (http://www.abridgegame.org/darcs/). Should do everything you need and has the advantage that you can create multiple repos for different purposes which all base on a single base repository. So i have a repository with all my config files and several others for different users which can pull their changes on top of the standard stuff. The syntax is quite easy to learn too.

Only drawback is that it is quite slow with really big repositories (e.g. linux kernel).

svn trick

[bluegreenone]

I use subversion (and tortoisesvn as the client) to version my windows system files, in general it works well. There is a trick though. To get a directory under svn control you normally have to import that directory and then rename it so you can checkout an svn'ed copy under the same name. This can be a problem for certain system directories. In this case what you do is :

1. create the dir in the repository but leave it empty
2. checkout that url on the existing dir, since url is empty nothing is overwritten
3. now do an svn add then commit to get everything into the repository

This leaves you with a versioned dir without need for renaming or deletion

...
PATH train [nynj.net] schedule online

Re:

[barzok]

Not sure I'd call something that's well-documented in the FAQ [tigris.org] a "trick."

Re:

[trawg]

Awesome trick! Very handy stuff.

This is my solution.. there are many like it,

[way2trivial]

this one is mine.
I use vice versa pro
http://www.tgrmn.com/ [tgrmn.com] disclaimer, I'm just a customer

It lets me sync files across different directories.. and if it finds changes, it has an option to archive a copy of what was there before in a third directory, with a date/time stamp.

you can set it to retain X# of copies, or for x# of days.. I have it set to automatically run on any change in the
mydocs folder on my pc at work, or my pc at home.. if I open a file at home and modify it, I can rest easy knowing the cha

CVS

[Colin Smith]

Config files are trivial to manage in CVS, it doesn't really need anything more sophisticated. You could probably get away with RCS but it'd be more hassle.

 

svn

[ManiaX Killerian]

As a lot of people suggested - SVN, one repository per type of server (so you can merge changesets even internally, although using diffs is also easy), the only problem we've got is that everyone uses root@ on that machine, and the repository is checked out by servername@svn-server, so everyone has to write his name when commiting changes. If it was possible to auth users based on ssh keys on the svn server, it would be perfect.

Re:

[penfern]

I do not prefer SVN over httpd (as their website seems to recommend). I always go for it straight through ssh. Much simpler and easier to manage, as long as all your users are already ssh users.

So yes, you can easily use ssh identities, or none at all and they have to type in their username/password every time. :)

Whatever you do

[JamesP]

don't waste your money on "Enterprise CVS solutions" that suck donkeys balls (the more expensive, the crappier it is, pretty much all of them).

They will gladly charge an arm and a leg for something as good (or worse) as CVS

Yes, I think you're thinking what I'm thinking and it is beyond me why people pay so much money for that crap. Probably because in their minds expensive == good

cfengine

[eviltypeguy]

One word: cfengine

http://www.cfengine.org/ [cfengine.org]

Re:

[nebulous_afterthough]

CFEngine is great for distribution and synchronization. It's not a version control system. In concert though, CFEngine with CVS is a great way to manage a lot of systems. You can even manage the CFEngine configuration within CVS for optimal control. Makes release management feasible and sane.

Re:

[doug]

I like that you're thinking outside of box. Every post I've read has tied replication with revision control. You're right that cfengine would do a great job of replicating the config files, but that doesn't remove the need for source management. At least one machine will still need rcs/svn/clearcase track the originals.

- doug

Break the problem down

[lamber45]

If you already have hundreds of heterogeneous systems, you don't want to add another authentication system, another logging system, and a whopping security hole. Better to see if existing tools can do the job, as much as possible.

I've been thinking about doing a project like this. I would probably use rdist+ssh/nfs/samba to copy the files to a central host, and run a cron job that calls a Perl script to check in changes regularly, only if something has changed; perhaps along with selections from

Easy.

[Slashcrap]

Simply write a script that runs MD5sum on every file in the filesystem once per minute and then copies any that have changed over the network to a central VMS server (which will take care of the versioning for you).

That's what you get for not listing cost effectiveness or efficiency amongst your requirements. Be more careful in future.

SVN for Configuration Management

[desertcrevasse]

First, some points about what would motivate a decision for Subversion:

- Revision control (Which includes all the things you do in software engineering: tagging, branching, merging, diffing, patching.)
- Excellent cross-platform support
- Several choices for client connectivity. HTTP(S)/mod_dav and SSH are the most popular.

That said, Subversion is NOT just for managing traditional software projects. There are many other types of controlled resources that require the features above. In our case (a large Uni

/etc under SVN control

[SCY.tSCc.]

About 2 years ago, we've introduced SVN to check in and keep control over our config files in /etc on our rootserver.

It allows us to see when, what and - given the administrator provided this information when the file was committed - WHY changes have been applied. That's pretty important in an environment, where up to 9 administrator have access to the files.

Saves you from losing precious configurations and allows for easy rollback if a certain configuration change doesn't work. You can even use your config

Re:

[commonchaos]

When I was thinking about doing something like this, my idea was to put the editor commands in a wrapper script which would launch the editor and do a 'svn ci' after the editor exits.