Ask Slashdot: Tools For Managing Multiple Serial Console Servers? 104
An anonymous reader writes "I've recently been charged with updating our existing serial console access tools. We have 12 racks of servers each with a console server in it (OpenGear, ACS, and a few others). Several of these systems host virtual machines which are also configured to have 'serial' management (KVM, virt serial). In total it comes to about 600 'systems.' All the systems also have remote power management (various vendors). Right now our team has a set of home grown scripts and a cobbled together database for keeping this all together. Today any admin can simply ssh into the master, run 'manage hostname console' and automatically get a serial console or run 'manage hostname power off' to cut the power to a system. I'd rather use some tools with more of a community than just the 4 of us. What tool(s) should I move my group onto for remote serial/power management?"
Conman and Powerman (Score:5, Informative)
Conserver (Score:4, Informative)
At a previous shop i worked we ran conserver against about 1k machines, some virtual but the big bulk of them was physical.
I also implemented a standardized way to control power on/off/reset on those machines as a patch to conserver. Those patches can be found at https://github.com/glance-/conserver/
Are you willing to monoculture your vendors? (Score:4, Informative)
Are you willing to monoculture your vendors?
If the answer is "no", then you are stuck with your home-grown stuff. Vendors intentionally introduce incompatibilities to lock you into using only them, so you aren't going to find some project that provides a HAL, or at least not one that will live through the next software update from one of your vendors.
You should also be aware (I'm sure you are, if you understand the dynamics of your scripts, but some reading this probably aren't) that some systems won't negotiate a KVM style console unless they are selected active in the KVM prior to boot, so there's an interaction between your power management sequencing and the virtual serial and real serial, and that varies from vendor to vendor and software update to software update.
If you are also using Real KVM(tm) style virtual video consoles, you're probably already aware that Linux and most other Open Source OS's fail to negotiate EDID information correctly, unless you use the closed source video drivers, unless they are selected as the active input on the virtual/real video display device, since those implementations are usually not multithreaded, and so if you have 4 HDMI inputs, and #2 is selected, and #4 is where the device is that comes up and does it's one-time negotiation (this is what's broken about the OS drivers: they should retry periodically until they get a response, then echo up the response to the video driver, which if it's in X/Wayland in user space, it's not going to happen, since it only happens at startup) you are SOL. So your scripts probably have that knowledge, too. Not that it'll do you any good if you have a Linux box on the second input on a physical Samsung monitor, mind you, as they automatically switch away from inactive inputs, and default tho the first input if there are none active.
So good luck with your ask, unless you are willing to start your own project, and are willing to push to get UEFI, u-boot, Linux/BSD video drivers, and other things fixed as part of the project overhead.
rtty, old school technology (Score:5, Informative)
There's a little known, but very useful program called rtty. You can find it at ftp://ftp.isc.org/isc/rtty/rtty-4.0.shar.gz. Yes, it was last updated in 2003. Yes, there are package for major open source distributions.
Here's serial consoles on the cheap:
Buy multiport USB to Serial devices. They are a USB hub with a bunch of USB to Serial adapters hung off of them. Here's a 16-porter for an example: http://www.startech.com/Cards-Adapters/Serial-Cards-Adapters/~ICUSB23216F
Hang them off a low end box, I like half-depth Intel Atom servers with lots of USB ports.
Run rtty. It records each console to a log file 24x7, and allows multiple people to connect at the same time (including typing).
Re:if it ain't broke... (Score:4, Informative)
My thought exactly. Unless they expect their console access needs to explode soon and the current system cannot scale, I can't see the need to change. The existing crew knows how to use the current setup and surely no more than a couple of pages of documentation would be all that's needed for any newcomer to come up to speed. Switching to a 3rd-party console access tool will just be one more thing that'll wind up appearing on the job adverts for new administrators and one more thing that'll slow down the hiring process when the HR filtering software doesn't see `ConsolePro2013++ Gold' on a candidate's resume.
Or did Management issue an edict of "Buy Not Build"?