Is There A Network Equivalent Of Alt-SysReq? 15
Random Q. Hacker asks: "Alt-SysReq has saved me from filesystem damage and runaway processes more than once. Unfortunately, several of the machines I admin are thousands of miles away in data centers, and it takes 15 minutes for data center personnel to go to our cage, hook up a monitor to the right system, and call back for interactive troubleshooting. I have played with snmpd, but it's a userspace daemon, and most of its functionality involves executing external programs and accessing files. Sometimes a system gets hung so bad (say, on root becoming unavailable, or memory becoming completely full) that the only thing still working is the kernel itself. Is there a kernel backdoor (as in a patch) that could let me have (secure, authenticated) SysReq functionality through the network?"
Not quite what you asked for, but ... (Score:3)
Another choice might be to insert a terminal server over at the remote end, connected to the lan on one side and the server's console port on the other ("10/100 Serial Server" over at Blackbox). It wouldn't give you a remote reset capability, but you'd be able to control the server no matter what state it was in, short of total unresponsiveness.
Might be a bit much to ask, but... (Score:1)
I'm speaking from only a general knowledge here, I'm not an expert on kernel hacking or anything, but possibly what could be done is this...
if you have multipul servers at the location, prehaps they could be connected via a serial port, and you could login to the other server that ISN'T completely ****ed up, and use the serial link? wether or not this will itself requier kernel patches I do not know, as I've never made use of a "serial console" in any way... but it seems a lot easier and probably more likely to happen than a secure authenticated method right to the locked server
just my $.02
I never knew what SysReq did anyway. (Score:1)
Re:Not quite what you asked for, but ... (Score:2)
I have also heard interesting things about the watchdog cards [ttp] that have Linux drivers in the kernel [linuxhq.com], although I've never used them myself.
Both these solutions strike me as ugly, but I guess it beats driving to a colloc in the middle of the night.
Lots of stuff. (Score:1)
Abstainer: a weak person who yields to the temptation of denying himself a pleasure.
Break (Score:2)
*Not a Sermon, Just a Thought
*/
Re:I never knew what SysReq did anyway. (Score:4)
What do you do with "SysReq"? Anything you want. On some systems, particularly in a "secure" environment, the "SysReq" key is how you get a login prompt because it is how you can ensure that you're seeing the real login program, not a password sniffing userspace front-end. That's the thinking behind WinNT using "Ctl-Alt-Del" to bring up the login screen.
On Linux, the kernel can be configured to bring up a very small "monitor" that allows you to perform a few tasks (e.g., sync'ing the hard disks and performing a clean shutdown) when all else fails.
I don't believe any handlers are installed for Windows non-NT or DOS.
Serial console (Score:1)
I've seen (awhile ago now) a company that makes a serial console 'card' for a linux machine. The machine sees it as an MGA or Hercules video adapter, and it allows *all* configuration over the serial port, including BIOS (Because the MB sees it as a video card). Once the kernel is booted, it switches to a standard serial mode, so you could even do things like "boot linux single" through lilo.
If I can find a link to the card again, I'll reply with it; but a search of Google might turn it up even.
Re:Serial console (Score:3)
Reports i'm hearing say it's absolutely fantastic, especially if you're on an non-top end server without serial-line BIOS availability. Downside? It's a bit pricy for the average user, but if you can afford a colo, you can probably afford one of these too...
Re:Serial console (Score:1)
That's the one! You could easily plug this into the server, configure the serial port for console use (mgetty, plus LILO config and kernel recompile for 100% serial console from bootup to shutdown).
And, since it can still emulate being at the 'console', you could send the Alt-SysRq keystroke, do what you must, and even tell the card to reboot the machine.
Re:Not quite what you asked for, but ... (Score:1)
1. a watchdog card. this card can be configured based on a number of things : processes running or not running - system heat - load etc. in the event that process x dies/hangs - you can have the card reset the machine.
2. pcweasel. this is a card that allows you serial connection to the box in any state at all - you telnet into it (that isnt bad. you put a secure box on a separate network. the you ssh into that box - then telnet into the serial port of the hung box. i know what you are going to say about telnet - but you just need design security into the setup) from this card you can even get to the bios functions of the machine and watch it the entire boot process. check it out - and demo it here: http://www.realweasel.com
which has been on
http://slashdot.org/askslashdot/99/06/17/173124
anyway - this is how i run many many boxes in remote colo's so i have to rely on the nocmonkey's as little as possible.
just be sure that you network layout takes into account that an additional backend network is needed for all maintainence tasks. needs additional NIC and also completely different IP space on the front end. the connection between the machines can be a private. then run BB on the front end firewall to watch all the crap going on with your boxen and report back to you.
or (Score:1)
Check out VACM (Score:1)
Zac
Re:I never knew what SysReq did anyway. (Score:1)
SSH and serial consoles (Score:1)
You could also have more than one SSH server, to have getty access through the second serial port in case the first SSH server is down.
You also can wire up remote RESET buttons, such as relays controlled through serial ports or "one-wire" controllers. But you're trying to avoid RESET so as to allow a more controlled shutdown.
Note that "man gdm" includes a feature which allows a mouse to run scripts -- such as making triple-click run a shutdown script. You can hang a mouse on a server so an obscure click sequence will do a controlled shutdown or restart...or wire the mouse port to a relay...
Also, as others have mentioned, hang a watchdog in there -- Linux includes a software watchdog but for a server you really should have a hardware watchdog card in there. You can also have more than one watchdog, with the first one initiating a software shutdown -- if that fails, the second one with a longer interval can automatically push the hardware RESET.