Preventing Shutdown on Active NFS Servers? 66
Ed Almos asks: "Like many Slashdot Readers, I run a small network at home with a server and a number of desktops. The server holds all our files as NFS shares and doubles as a desktop machine should the need arise. Problems however occur if the server is shut down whilst there are NFS shares in use, the minimum disruption is a crashed desktop and a couple of times I have had to deal with corrupted files. Does anyone know of a way to prevent shutdown of a machine if someone else has drives mounted to its NFS shares ? I have already explored use of the /etc/shutdown.allow file but all this does is determine who can kill the machine. The minimal solution would be something similar to a Microsoft Windows system, where a request to shutdown brings up a warning window that there are users connected to the system, but I am not sure how to achieve this on a Linux system. Ideally I would like to prevent shutdown of a system with active NFS shares altogether, or at least until the user has unmounted and logged off the network."
Can't do it (Score:5, Insightful)
NFS is stateless from the server's perspective. This is done so that the server doesn't have to track the state of a whole fleet of clients (and so that the server can pick up where it left off when it crashes and restarts).
So the server, by design, has no notion of the number / names of users connected to it.
The best you could do would probably be to monitor NFS traffic, and present a dialog on shutdown if there has been any traffic in the last 5 minutes or so.
Implementation (Score:2, Interesting)
Re:Implementation (Score:5, Insightful)
Want to see your uptime and stability rise incredibly on the server? Put it in the closet on a UPS and once it is running turn off the monitor, unplug the keyboard, and tape a piece of cardboard over the power switch so it doesn't get turned off by accident. Where the machine used to sit put a cheap replacement computer to use as a workstation - even new entry level boxes are starting at under $500 fully loaded (a little wimpy, but including all the necessary parts including a monitor) and used hardware has gotten insanely cheap (ie $200 for a full machine that is a generation or two old, PIII
That said, I am going to read every post in this thread to get a better understanding of how to do this - now you have my interest up.
Tsk Tsk (Score:1)
Re:Tsk Tsk (Score:1)
Besides, checking to see if $DISPLAY is set tells you if the command was run from within an X display.
Yeah, but the client thinks it's stateful (Score:5, Informative)
mount | grep $1 | wc -l
then write a wrapper on the server that does
foreach client (client list)
mounted = ssh client nfsounts `hostname`
ok = false if (mounted)
end
you can hook that into your shutdown script, and abort if there are any clients who think they have a mounted drive.
of course, read the other suggestions about mount options. Noone's mentioned sync yet, but don't mount your shares async even though the performance is so much better or you'll loose data.
Re:Yeah, but the client thinks it's stateful (Score:1)
And if a machine dies (think "power failure" or "kernel panic") before decrementing the mounted count file? The grandparent's solution doesn't have that limitation, in exchange for a little processor time and a little bandwidth.
Re:Can't do it (Score:1, Informative)
/var/lib/nfs
/var/state/nfs
which specificly tell IP addresses and mountpoints of clients. If those files are deleted/modified while nfs is off, when nfs returns it won't know anything about the clients already connected, and their accesses will fail.
Re:Can't do it (Score:1)
Wrong - the nfsd is statless.. mounts, locks and stuff like that persists...
Re:Can't do it (Score:2)
NFS is indeed stateless, however the server does know of users "connected" to it via the mount RPC protocol which is stateful. Try 'showmount'.
But... (Score:3, Informative)
Theoretically, once the NFS server has crashed, shouldn't all clients simply freeze until the server is back? On all systems I used, this was the observed behavior, and it is quite useful actually: it seems to avoid all data loss problems (under conditions). When the NFS gets reachable, all running program go on executing as if nothing had happened.
A solution to the original problem, though, would be: tell all user that the NFS machine is to be powered on constantly.
Re:But... (Score:2, Informative)
low tech solution (Score:3, Interesting)
2) Put your NFS server on the *best* machine, not the worst one, so that users want to use it first. If worst comes to worst, put signs on the other machines advertising the other's superiority. (and without the NFS overhead, it will really outperform its clients!)
3) Put your NFS server on a g3 laptop or other ultra low power system, and hide the system in your closet so others can't find it to turn it off. (hardware suggestions, anyone?)
4) switch to samba (heh heh heh)
Re:WARNING: FIRE HAZARD!!!!! (Score:1)
Re:WARNING: FIRE HAZARD!!!!! (Score:3, Funny)
Heck, you could even build some kind of coolstore. A bit OTT but doable if you have the money.
Re:WARNING: FIRE HAZARD!!!!! (Score:2)
Besides, if you put your electronics in a poorly ventilated space and a small fire starts, it'll smolder and then die instead of flaring up."
One of the reasons that drywall (gypsum board) is so heavy is that there's about a gallon of water in every 4 square feet so by the time you can get any more of it than the paper backing to burn
You've got something wrong... (Score:4, Informative)
NFS is explicitly designed to be stateless, precisely to allow it to function across server reboots, crashes, and other fun. If your clients are crashing, or getting back corrupted data, something is screwed up somewhere.
And, by the way, if you're getting corrupted data on a server crash, and the server is linux, you just had an object lesson on why it's bad that linux NFS defaults to async writes.
Re:You've got something wrong... (Score:1)
"In releases of nfs-utils upto and including 1.0.0, this option was the default. In this and future releases, sync is the default, and async must be explicit requested if needed."
From the rpm changelog:
"* Mon Jul 22 2002 Bob Matthews bmatthews@redhat.com>
- Move to nfs-utils-1.0.1"
'nuff said.
May be wrong... (Score:1)
Re:May be wrong... (Score:2)
Try rwall or similar (Score:4, Informative)
If that isn't good enough for you, there are a couple other possibilities. You could probably cobble together an utterly trivial Python (or Perl or whatever) script on your client machines, then have the server invoke it via ssh when a shutdown starts. If you aren't a programmer at all, you could try firing off an email to the client machines. As long as you have a periodic mail-checker going, it would alert you to the arrival of a new message. (Since you'd be able to use the local spool, you could have it check every 15 seconds.)
Replace /sbin/shutdown with a wrapper script... (Score:2, Interesting)
a) Move
b) Write a shell script called
If you want to make it really fancy, you can do something like calling and parsing the output of 'df', comparing that to the contents of
Re:Replace /sbin/shutdown with a wrapper script... (Score:1)
I just read a post below mine and realized I *completely* misunderstood the question.
Nevermind!
Message From Original Poster (Score:1)
Ed
Re:Message From Original Poster (Score:1)
use correct mount options (Score:5, Interesting)
The options you want (for filesystems mounted rw) are:
rw,hard,nointr...
A lot of people don't like these options because it means that the clients will hang until the server returns, but it is THE RIGHT THING TO DO if you are mounting important data rw. If you can't stand for your clients to hang, maybe replace 'nointr' with 'intr', but you've been warned.
Steve
Mod parent up (Score:2, Informative)
Re:Mod parent up (Score:4, Interesting)
I believe that it is theoretically possible to write software that can survive a soft mounted filesystem disappearing from under it, but no one ever does. How often do people check the return value from write()? And in memory mapped io land, it would be nasty.
Re:use correct mount options (Score:1, Redundant)
rw,hard,nointr
Those are the correct options for a read-write NFS drive that will freeze the clients until the server has been restarted. After restart the clients continue as if nothing has happened.
Re:use correct mount options (Score:2)
Hey, this IS an Ask Slashdot, right? :-)
Re:use correct mount options (Score:2)
Re:use correct mount options (Score:2)
But doesn't this cookie system sorta break the statelessness of the NFS server, at least in spirit? What's the reason for doing it this way instead of having the client ask for full path names? Can this behaviour be turned off somehow? Having th
Re:use correct mount options (Score:2)
Sorry, dude.
maybe... (Score:4, Interesting)
Then write a wrapper around each of halt, shutdown, and reboot to check the open ports and fail if they are active.
Seems fairly hackish, but... whaddya expect from
NFS Locking Service might work? (Score:5, Interesting)
NFS input/output is stateless, but I believe the locking mechanism is stateful.
When clients are accessing a file, a lock is established. When the client is done with the file, the lock is removed. You can see who has what resource locked with a utility (I forget which, but fcntl() and lockf() come to mind).
In a shutdown script, look for locks, and refuse to procede until the locks are cleared.
Re:NFS Locking Service might work? (Score:1)
Sorry, that's "NFS Locking Services"
NFS Locking is the cat's meow. (Score:2)
Try this:
Create
After the clients mount the nfs drive, have them perform a read (non-exclusive) lock on
(clientnfslock
In the server shutdown script put in a routine that fails if it can not perform a write (exclusive) lock on that same file.
(... servernfslock
Note
maybe i'm just stupid.. (Score:4, Insightful)
look, what point would there be in initiating the shutdown if you didn't know when the users will get out anyways? it could take hours/days before it would actually boot, and if that doesn't matter(waiting _hours_) then why would you be booting it in the first place? just out of habit?
anyways.. it sounds a lot like you should be fixing on why you have to be booting it instead of how the booting occurs.. so that you wouldn't need to be booting it.
Alternate solution (Score:3, Interesting)
Couple suggestions.... (Score:2, Informative)
Second, use FTP instead of NFS. Allow it to support resume, only let it talk locally. I believe there are utils which will allow you to mount an FTP as a drive in windows as well...
couple of ideas anyways.
NFS Automounter (Score:2, Interesting)
Re:NFS Automounter (Score:2, Insightful)
At larger sites, you also get some other great benefits, like being able to move filesystems transparently from one server to another without touching the clients, all by managing the automounter map.
Most Unix and Linux variants have a standard automounter. Use it! This will lead to two possible scenarios when the fileserver
Mod parent up (Score:2)
rmtab (Score:3, Interesting)
1. When shutting down, first go through rmtab and send an rwall message to those machines, saying 'get the heck off because the server is going down shortly'
2. Two minutes later try again and send a more forceful message.
3. Two minutes later tell them they are about to get it in the neck. And shutdown.
If you really dont want to shut the machine down with NFS mounted stuff still there, then modify to taste - dont shutdown unless rmtab doesn't have any shares from machines you dont want to annoy.
This is untested of course.... Until you try it! Read the warnings about rmtab in man mountd. It may not be trustworthy.
Baz
Simple ... (Score:1)
Solved a similar problem ... (Score:2, Informative)
... a long time ago. Basically, every user gets a simple switch, all switches are connected in parallel, so you get a "wired-or". All users are told: If you need the server, turn on that switch. If you don't need the server any more, switch it off again. A little piece of electronic connected to a RS232 port and a tiny C programm control the power supply for the server. Details (german only, sorry) are here [foken.de].
The trick is: The server now knows best when to start and to shutdown, there is no more need for man
lsof ? (Score:1)
Home network server (Score:2, Informative)
Similar setup:
one Linux file server, 3 desktops (mixed nix and win2k), when the server shuts down, any cients active go nuts (winamp freezes, explorer complains about the sudden disconnect)
best theing to do, since you're on a home network is to make sure no one's using the damn thing before you shut it down.
This is easy, even with 6 systems, (assuming samba), just make sure no programs are activly using the shares and reboot the box, the wor
showmount anyone? (Score:2, Informative)
Best Solution (Score:1)
Put a sign on the server that reads "Do not shutdown this server". If you want a technical solution - use a post-it.