Is There A Network Equivalent Of Alt-SysReq?

Random Q. Hacker asks: "Alt-SysReq has saved me from filesystem damage and runaway processes more than once. Unfortunately, several of the machines I admin are thousands of miles away in data centers, and it takes 15 minutes for data center personnel to go to our cage, hook up a monitor to the right system, and call back for interactive troubleshooting. I have played with snmpd, but it's a userspace daemon, and most of its functionality involves executing external programs and accessing files. Sometimes a system gets hung so bad (say, on root becoming unavailable, or memory becoming completely full) that the only thing still working is the kernel itself. Is there a kernel backdoor (as in a patch) that could let me have (secure, authenticated) SysReq functionality through the network?"

Not quite what you asked for, but ...

[John Jorsett]

I've had similar problems in the past, though in my case it was that I didn't have security clearances high enough to get into the facility where the machine I developed resided. What we ended up doing was to set up an encrypted phone line on which I could dialup the remote system and connect to the console port. In my case, if the system had to be reset, I had a user at the other end do it. However, there are devices which allow you to do power cycling or equipment reset via the telephone (there's one called Power Mate over at Blackbox Catalog [blackbox.com]). If your remote site can dedicate a phone line to the equipment, an encrypted modem call and a remote reset capability might be the way to go.

Another choice might be to insert a terminal server over at the remote end, connected to the lan on one side and the server's console port on the other ("10/100 Serial Server" over at Blackbox). It wouldn't give you a remote reset capability, but you'd be able to control the server no matter what state it was in, short of total unresponsiveness.

Might be a bit much to ask, but...

[NNKK]

Secure authenticated? that might be a bit much to ask.... however...
I'm speaking from only a general knowledge here, I'm not an expert on kernel hacking or anything, but possibly what could be done is this...
if you have multipul servers at the location, prehaps they could be connected via a serial port, and you could login to the other server that ISN'T completely ****ed up, and use the serial link? wether or not this will itself requier kernel patches I do not know, as I've never made use of a "serial console" in any way... but it seems a lot easier and probably more likely to happen than a secure authenticated method right to the locked server

just my $.02

I never knew what SysReq did anyway.

[AFCArchvile]

What does SysReq do? It seems to do nothing in Windows and DOS, but what does it do in Linux?

Re:Not quite what you asked for, but ...

[dubl-u]

I'm also not quite sure what the original poster wants, but when a box in a cabinet gets so unhappy that it's not controllable via the network, I use APC's MasterSwitch [apcc.com], which is basically and eight-plug power strip with an ethernet jack and the ability to turn the outlets on and off individually.

I have also heard interesting things about the watchdog cards [ttp] that have Linux drivers in the kernel [linuxhq.com], although I've never used them myself.

Both these solutions strike me as ugly, but I guess it beats driving to a colloc in the middle of the night.

Lots of stuff.

[yogensha]

There are lots of things you can do really. APC and others make products that allow you to power cycle machines remotely. I've seen 1u boxes with a bunch of serial ports that allow you to access consoles remotely. You could also build a box with a bunch of serial ports and connect consoles that way. I have various networking devices connected to serial ports on a FreeBSD box, and it's great to console into them when you hosed something :)


Abstainer: a weak person who yields to the temptation of denying himself a pleasure.

Break

[The Madpostal Worker]

If its PC hardware, sending a break over the serial port does the same as SysRq, so if you got a console server(these let you use serial ports over the network, you usually telnet into them and then pick a port) you could do all the Magic SysRq stuff remotely... Combine this with a somthing to do remote poweroffs, a modern bios that does serial console stuff (lets you config the bios), a properly configured bootloader, and a properly configured kernel and you'll never need a physical console (excluding hardware failures).

/*
*Not a Sermon, Just a Thought
*/

Re:I never knew what SysReq did anyway.

[coyote-san]

The easiest way to answer this is to compare it to the "break" signal on serial ports, or the hook "flash" on telephones. The SysReq key is (or is supposed to be) an equivalent "out of band" signal from the keyboard and should always be recognized even if the keyboard buffer is full, hosed, or otherwise unusable.

What do you do with "SysReq"? Anything you want. On some systems, particularly in a "secure" environment, the "SysReq" key is how you get a login prompt because it is how you can ensure that you're seeing the real login program, not a password sniffing userspace front-end. That's the thinking behind WinNT using "Ctl-Alt-Del" to bring up the login screen.

On Linux, the kernel can be configured to bring up a very small "monitor" that allows you to perform a few tasks (e.g., sync'ing the hard disks and performing a clean shutdown) when all else fails.

I don't believe any handlers are installed for Windows non-NT or DOS.

Serial console

[srhuston]

Along the lines with those who suggested a serial console, perhaps hooked to a terminal server:

I've seen (awhile ago now) a company that makes a serial console 'card' for a linux machine. The machine sees it as an MGA or Hercules video adapter, and it allows *all* configuration over the serial port, including BIOS (Because the MB sees it as a video card). Once the kernel is booted, it switches to a standard serial mode, so you could even do things like "boot linux single" through lilo.

If I can find a link to the card again, I'll reply with it; but a search of Google might turn it up even.

Re:Serial console

[awx]

Ahh, that would be the RealWeasel: http://www.realweasel.com/intro.html

Reports i'm hearing say it's absolutely fantastic, especially if you're on an non-top end server without serial-line BIOS availability. Downside? It's a bit pricy for the average user, but if you can afford a colo, you can probably afford one of these too...

Re:Serial console

[srhuston]

Ahh, that would be the RealWeasel: http://www.realweasel.com/intro.html

That's the one! You could easily plug this into the server, configure the serial port for console use (mgetty, plus LILO config and kernel recompile for 100% serial console from bootup to shutdown).

And, since it can still emulate being at the 'console', you could send the Alt-SysRq keystroke, do what you must, and even tell the card to reboot the machine.

Re:Not quite what you asked for, but ...

[Anonymous Coward]

another thing you can look into (in addition to these comments above) is this:

1. a watchdog card. this card can be configured based on a number of things : processes running or not running - system heat - load etc. in the event that process x dies/hangs - you can have the card reset the machine.

2. pcweasel. this is a card that allows you serial connection to the box in any state at all - you telnet into it (that isnt bad. you put a secure box on a separate network. the you ssh into that box - then telnet into the serial port of the hung box. i know what you are going to say about telnet - but you just need design security into the setup) from this card you can even get to the bios functions of the machine and watch it the entire boot process. check it out - and demo it here: http://www.realweasel.com

which has been on /. b 4. here:

http://slashdot.org/askslashdot/99/06/17/1731248 .s html

anyway - this is how i run many many boxes in remote colo's so i have to rely on the nocmonkey's as little as possible.

just be sure that you network layout takes into account that an additional backend network is needed for all maintainence tasks. needs additional NIC and also completely different IP space on the front end. the connection between the machines can be a private. then run BB on the front end firewall to watch all the crap going on with your boxen and report back to you.

or

[ArchieBunker]

Get a real unix box. Any true unix box will default to using the first serial port when no keyboard is present. This holds true for Sun/HP/DEC/SGI/.

Check out VACM

[zacs]

VACM [sourceforge.net] can provide all the functionality you're looking for and more... It provides the ability to monitor machines at both the hardware and software level.

Zac

Re:I never knew what SysReq did anyway.

[Anonymous Coward]

SysReq was in IBM thing. It's still used in the S/390 and as/400 world (oh excuse me, iSeries400 now) to get a "System Request" menu to do various things, like kill jobs, switch systems, etc.

SSH and serial consoles

[Anonymous Coward]

Right. Have an SSH server with a bunch of serial ports. Have all the other servers configured to use serial consoles. Those serial consoles all wired to the SSH server. That will give you secure remote non-networked access to all your other servers (the "non-networked" part avoids problems with the network drivers having failed).

You could also have more than one SSH server, to have getty access through the second serial port in case the first SSH server is down.

You also can wire up remote RESET buttons, such as relays controlled through serial ports or "one-wire" controllers. But you're trying to avoid RESET so as to allow a more controlled shutdown.

Note that "man gdm" includes a feature which allows a mouse to run scripts -- such as making triple-click run a shutdown script. You can hang a mouse on a server so an obscure click sequence will do a controlled shutdown or restart...or wire the mouse port to a relay...

Also, as others have mentioned, hang a watchdog in there -- Linux includes a software watchdog but for a server you really should have a hardware watchdog card in there. You can also have more than one watchdog, with the first one initiating a software shutdown -- if that fails, the second one with a longer interval can automatically push the hardware RESET.