Ask Slashdot: Tools For Managing Multiple Serial Console Servers? 104
An anonymous reader writes "I've recently been charged with updating our existing serial console access tools. We have 12 racks of servers each with a console server in it (OpenGear, ACS, and a few others). Several of these systems host virtual machines which are also configured to have 'serial' management (KVM, virt serial). In total it comes to about 600 'systems.' All the systems also have remote power management (various vendors). Right now our team has a set of home grown scripts and a cobbled together database for keeping this all together. Today any admin can simply ssh into the master, run 'manage hostname console' and automatically get a serial console or run 'manage hostname power off' to cut the power to a system. I'd rather use some tools with more of a community than just the 4 of us. What tool(s) should I move my group onto for remote serial/power management?"
How about SourceForge? (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
What does "a community" have to do with whether the tools work or not?
To Quote - " I'd rather use some tools with more of a community than just the 4 of us."
He also never said that there were shortcomings in the toolset they created. It sounds like he may not like the database, maybe he wants a nicer front-end for managing the tables? But it's not described as 'the problem'.
Therefore, if they create a community around their own toolset, then the only problem actually described in the OP is resolved.
Re: (Score:1)
Re: (Score:2)
Well, maybe not SourceForge [slashdot.org] specifically, but yeah.
Switch to a cloud system ... (Score:1)
... like OpenStack. Then you have access to everything.
Re: (Score:2)
IPMI can certain help at the hardware level. But if you need frequent hardware level access, or access to it by more than a couple people, something is wrong. The cloud infrastructure allows appropriate virtual level access (terminate an instance and start a new one). Maybe his "community access" was meant to do OS installs, which an infrastructure cloud stack would do. If not, maybe we need to know what the OP is really wanting to do.
Re: (Score:2)
Use the current host hardware.
Why? (Score:5, Insightful)
You haven't identified any missing features or existing anti-features in your current toolset.
The only hint of anything wrong with your setup is "I'd rather use some tools with more of a community than just the 4 of us."
Q: What tool(s) should I move my group onto for remote serial/power management?
A: Yours. Clean the tools up, open source them, and market them. Your community will grow.
Try Digi (Score:1)
You might look at Digi.
I've used their stuff and it works well.
Conman and Powerman (Score:5, Informative)
Re: (Score:1)
Seconded. I use these every day.
Re: (Score:2)
Thirded. Conman & powerman rock.
Re: (Score:1)
Fourthed; again, use almost every day. Conman and powerman are great.
Conman: Serial consoles, IMPI serial consoles, write your own wrapper if one doesn't exist.
Powerman: Can control just about everything out there. Again, write your own if it doesn't exist. Can even "manage" power on a VM host. You might have to write your own wrapper though (basically a remote shell that runs virsh commands). I know we have a script here, but I don't think I'm allowed to put it in public. It uses rsh to connect to the VM
Conserver is your answer (Score:3, Funny)
I would recommend a look at http://conserver.com/
Re:Conserver is your answer (Score:4, Insightful)
I actually was about to post that....with one caveat: Look carefully at versions
Admittedly I haven't worked with conserver since 2005, but it was solid then and I can't imagine it has changed much since. The last time I grabbed it, I found that the source had forked a couple of times and the name "conserver" actually refered to 2 or three different versions of the same program, each with slightly different feature sets.
Few things are more frustrating than trying to figure out why a program isn't working because you are reading the docs for the wrong one.
That said, the conserver we ended up using was simple, lightweight, and did exactly what we wanted.... provide named console access from a single place.
Re: (Score:2)
Conserver is great. I've used it to monitor Linux consoles after boot (via grub handoff to serial console). Serial consoles are cheaper per port then KVM port and you have a log you can grep.
I've also used it to monitor several consoles going to embedded devices. The users could take over when a coworker had gone on vacation w/o calling the sysadmin.
Conserver (Score:4, Informative)
At a previous shop i worked we ran conserver against about 1k machines, some virtual but the big bulk of them was physical.
I also implemented a standardized way to control power on/off/reset on those machines as a patch to conserver. Those patches can be found at https://github.com/glance-/conserver/
Uplogix (Score:2)
My company builds a replacement (http://uplogix.com), smarter version of a serial console. They can all be managed by web UI and you can term directly into each device, keep configuration on them, and keep each device mapped to its outlet on the power controller. We even have a virtual version that runs on vSphere. You can hook up all the ports via telnet and keep your existing term server, but getting the benefits of the rich CLI and web UI.
Sounds like a perfect use case for you.
Re: (Score:2)
WebUI? WHY IN THE NAME OF THE FSM would any sysadmin want that?
People who want serial console access are not looking for "teh shiny".
handle any fan hit anywhere, anytime (Score:2)
A web UI means I can handle an issue from any device - a computer in a hotel, grandma's cell phone, whatever.
Also, with many systems, a UI to navigate groups of systems can be handy.
Re: (Score:2)
Just use SSH. You can even keep a copy of a client for most OS with you.
I see you didn't read a word before replying (Score:2)
I see you didn't bother to read one word of my post before replying.
Does the computer in the hotel business center have ssh installed? No. Does a borrowed phone have ssh? No.
Ssh is great, until I run out of battery.
Re: (Score:2)
A thumb drive does not need batteries.
You can use an ssh client without installing it from a thumbdrive.
You can install a free ssh client on that phone.
When I am on call I always carry my charged external battery pack.
try that and tell me how it works (Score:2)
Try running some random executable from a thumb drive on a hotel computer and tell me how will that works.
Then the next person you see, ask them if it's okay for you to install weird "hacker" apps on their phone
Re: (Score:2)
I do not work from either of those. Do you really trust neither have key loggers installed?
Re: (Score:2)
Re: (Score:2)
So, wait, you replace SSH on a known secured computer (at least I hope an admin's travel computer is relatively secured) with a web UI? So you can use it from a adware, spyware filled device like a hotel lobby computer or grandma's cell phone filled with spying, keylogging games? Sure, the web UI might be over HTTPS, but that does nothing about spyware seeing you punch in the URL, then type in your username and password. I really hope your IDS knows when you are traveling, and will use the website, and when
good point. retracted. For not security sensitive (Score:2)
That's a good point and I retract my comment in the context of console servers.
The point I had in mind is that although I use CLI for almost everything, sometimes a GUI is much nicer. The CLI for LSI RAID cards comes to mind.
Re: (Score:2)
Locally, sure. Heck, dropping a UI in front of the scripts that the article was asking about might make the management of the variety of devices easier; Just toss together a python program, or other easy language at hand, with the ability to call bash scrips and the ability to throw up a UI and mess with the database that tracks all the devices and their login details; that would make their scripts more usable should they all get fired tomorrow and the new team has access to an easier work flow.
But remote a
Re: (Score:2)
If you're using the computer in the hotel business center to get console access, you probably don't care about security. If you care about security at all, you're going to use your own device.
Re: (Score:2)
Re: (Score:3)
The advanced CLI is for the sysadmin. The WebUI allows you to lock down users to say 1 port on a machine and give them a nice shiny button to click on.
if it ain't broke... (Score:5, Insightful)
Re:if it ain't broke... (Score:4, Informative)
My thought exactly. Unless they expect their console access needs to explode soon and the current system cannot scale, I can't see the need to change. The existing crew knows how to use the current setup and surely no more than a couple of pages of documentation would be all that's needed for any newcomer to come up to speed. Switching to a 3rd-party console access tool will just be one more thing that'll wind up appearing on the job adverts for new administrators and one more thing that'll slow down the hiring process when the HR filtering software doesn't see `ConsolePro2013++ Gold' on a candidate's resume.
Or did Management issue an edict of "Buy Not Build"?
Re: (Score:2)
My thought exactly. Unless they expect their console access needs to explode soon and the current system cannot scale,
Agreed. Standardization should decrease complexity (of maintenance), not increase it. Keep the tools as simple as possible, and as close to vendor standard as possible. It's like admins who customize root. If you leave it generic, then every admin knows what to expect.
Re: (Score:2)
It's like admins who customize root. If you leave it generic, then every admin knows what to expect.
... and every intruder too!
Re: (Score:2)
Now *how* would an intruder get root?
Sarcasm, aside...
There are benefits to keeping things standard and as long as you implement reasonable security configurations and don't allow non-local root logins, I don't see any real reason to change root in most cases. Your life (and the lives of admins to come behind you) will be much easier.
Re: (Score:3)
I completely agree except for one thing.... and I am looking at YOU redhat..... if the system default that they come to expect is that ls output is unreadable due to dircolors being the distribution default which assumes a light colored background.... that should be fixed with extreme prejudice.
Re: (Score:2)
when the HR filtering software doesn't see `ConsolePro2013++ Gold' on a candidate's resume.
Hmmm...and I am only experienced with ConsolePro2011++ Gold...time to upgrade the skillset...
Re: (Score:2)
Re: (Score:2)
Because virtual server cloud crap is all about resume buzzwords, brah.
It doesn't matter if you've produced an excellent in-house solution, because THAT DOESN'T GET YOU A JOB ELSEWHERE. Much better to switch to something which'll end up with your spending 50% of the original time adapting to the new system, and another 80% implementing all the missing features.
Re: (Score:2)
Obviously, you want to change it... because? Jeez.
I think OP is trying to address the issue of having an infrastructure running on a bunch of home grown stuff supported by just a couple guys. If it works, great but when one or two or three of those guys leave, all of a sudden nobody knows how the thing works and you have a mess. If there is a mandated documentation ritual along with revision control, it helps, but that's probably not the case. OP is wondering if there is an open source solution out there already doing what they've duct-taped together. In
Re: (Score:2)
No kidding. Large enterprises are built from cobbled together scripts and databases.
Are you willing to monoculture your vendors? (Score:4, Informative)
Are you willing to monoculture your vendors?
If the answer is "no", then you are stuck with your home-grown stuff. Vendors intentionally introduce incompatibilities to lock you into using only them, so you aren't going to find some project that provides a HAL, or at least not one that will live through the next software update from one of your vendors.
You should also be aware (I'm sure you are, if you understand the dynamics of your scripts, but some reading this probably aren't) that some systems won't negotiate a KVM style console unless they are selected active in the KVM prior to boot, so there's an interaction between your power management sequencing and the virtual serial and real serial, and that varies from vendor to vendor and software update to software update.
If you are also using Real KVM(tm) style virtual video consoles, you're probably already aware that Linux and most other Open Source OS's fail to negotiate EDID information correctly, unless you use the closed source video drivers, unless they are selected as the active input on the virtual/real video display device, since those implementations are usually not multithreaded, and so if you have 4 HDMI inputs, and #2 is selected, and #4 is where the device is that comes up and does it's one-time negotiation (this is what's broken about the OS drivers: they should retry periodically until they get a response, then echo up the response to the video driver, which if it's in X/Wayland in user space, it's not going to happen, since it only happens at startup) you are SOL. So your scripts probably have that knowledge, too. Not that it'll do you any good if you have a Linux box on the second input on a physical Samsung monitor, mind you, as they automatically switch away from inactive inputs, and default tho the first input if there are none active.
So good luck with your ask, unless you are willing to start your own project, and are willing to push to get UEFI, u-boot, Linux/BSD video drivers, and other things fixed as part of the project overhead.
Re: (Score:2)
WTF are you talking about?
There are plenty of open source tools that can support lots of vendors. Conman is just the first one I thought of.
By KVM he means Kernel Virtual Machine not Keyboard Video Mouse.
rtty, old school technology (Score:5, Informative)
There's a little known, but very useful program called rtty. You can find it at ftp://ftp.isc.org/isc/rtty/rtty-4.0.shar.gz. Yes, it was last updated in 2003. Yes, there are package for major open source distributions.
Here's serial consoles on the cheap:
Buy multiport USB to Serial devices. They are a USB hub with a bunch of USB to Serial adapters hung off of them. Here's a 16-porter for an example: http://www.startech.com/Cards-Adapters/Serial-Cards-Adapters/~ICUSB23216F
Hang them off a low end box, I like half-depth Intel Atom servers with lots of USB ports.
Run rtty. It records each console to a log file 24x7, and allows multiple people to connect at the same time (including typing).
Re: (Score:2)
Problem with this is that every time you plug in a new USB device, you are likely to totally hose the device names on the next reboot. It is *really* hard to tell what order your USB devices will be initialized and in this case it will be a serious issue trying to sort out what port is which every time you want to add or remove a device.
Re: (Score:2)
It's pretty easy to find USBSerial interfaces that include a unique ID of some sort. That plus 4 seconds editing your udev config will give you stable device naming.
Re: (Score:2)
As I understood this suggestion, one was to go pickup a pile of identical USB to serial converters and USB hubs and just wire them all up. There will be nothing unique between the devices except how they are connected to the host USB hub and in which order the devices are initialized.
Re: (Score:2)
I've not seen a hardware/software combination in the past ~5 years that varied the initialization order. Now, the order in which the USB ports are initialized is often non-intuitive and non-documented, but just plug a flash drive into each available port and reboot the server to find the order they are initialized in.
Where I see most people go wrong is they don't figure out the order the ports are initialized, and thus don't plug the serial device into the first to be initialized. By plugging into some ot
Re: (Score:2)
Which makes my point. I'm not saying that the order is not set, but that anytime you make a change to the hardware configuration by plugging something in or removing a device you may unknowingly move a whole bunch of stuff in a way you don't understand. This might force you to have to go back though 120+ devices per host port and sort out what port is which device, just because something changed.
Consider what happens if one of your first 50 devices fails (say it comes unplugged or the cable breaks). What
Re: (Score:2)
I think you misunderstood what I said.
The OS (at least, Linux and FreeBSD) enumerate the motherboard ports in order. So if your motherboard has 8 ports you can discover the order and number them 1-8.
If you plug in a device to port 1, and then later plug the next device into port 2, there is no chance the first device will be renumbered.
However, if you plug into port 2, and later add a device on port 1, it will get renumbered.
So when I get a new motherboard, I take a pile of old 16M (yes, Meg) flash drives
Re: (Score:2)
Problem with this is that every time you plug in a new USB device, you are likely to totally hose the device names on the next reboot.
Look at /dev/bus/usb. Unless you lose a USB controller those device names should be stable.
Establish priorities first! (Score:2)
The most important thing is that it works reasonably well and doesn't require excessive administrative overhead. You've described a reasonably well managed configuration, so it's tough to justify changing everything "just because".
Exactly what would you accomplish by switching to an "open" technology? Answer that, and then you can make the best decision.
HP and Dell have "free" tools (Score:2)
to manage their servers including console access via a special NIC port and the ability to push the power button remotely through a web browser
basic features are free. remote console is a $400 per server option called iLO Advanced on HP servers. the management software is free as well with some features licensed per server
Re: (Score:2)
Beware the evils of OOB management. Much of the time, that functionality in a PC server is handled by an IPMI module, which is a little computer in its own right which will probably never receive adequate security patches and which has access to twiddle important configuration values in your system. They're wicked cool when they're not being used to own your network, though.
This is what I'm hearing (Score:2)
"We have something that works and I want to break it"
Re: (Score:2)
Create your own community (Score:2)
Livingston PortMasters (or similar) (Score:1)
At $WORK we have used Livingston PortMasters to deal with our serial console issues. Network connected and logins can be setup to only have access to certain ports. Have worked well for us. Usually found pretty cheap on eBay.
Re: (Score:2)
Re: (Score:1)
Actual implementation for us is also SSH'ing to a *NIX box, then tipping over to the PortMaster.
Re: (Score:2)
When we got more powerful hardware for the SSH bastion host, I wrote a set of daemons and scripts which would maintain one 'screen' session for each console port. At startup it would enable logging, make a 'telnet' connection, and then disconnect the session and leave it idle. When a user wanted to access a port, they'd run a menu tool (setuid launcher and Perl, iirc) that would give them a
MRV (Score:2)
Re:Please help me understand (Score:4, Insightful)
Re: (Score:2)
Outside the realm of windows, for boot and boot issues you'll be in command line land anyway. Most mainframe and supercomputer and midrange and micros in my 30+ years of professional life other than windows, OS/2, later novell netware...all needed the command line skills.
If it is not broke.. (Score:2)
If the current system works well then don't try to fix it. " I'd rather use some tools with more of a community than just the 4 of us." is a very bad reason to try to install and learn a completely different system.
Unless more information is presented, I'll place this in the "idle hands are the devil's playground" category.
Re: (Score:2)
Well one aspect to consider is, keeping a bunch of staff tied into using some proprietary internally developed tools could also be isolating those staff from gaining any experience with current tools out there in the market and in use by many/most future employers. I am stuck in a situation like this myself, where any chance to get a new job will rely on me learning new development tools in my own time on my home PC, then trying to convince somebody that this is the same as workplace experience with the to
Re: (Score:2)
Re: (Score:1)
Consoleworks is what we used at one of my previous employers. It had software that would run on Windows or Linux.
It'all there! Why don't you use it? (Score:1)
Disclaimer: I work for these guys: http://www.ovirt.org/Features/DRBD [ovirt.org]
As somebody said before, this shop sounds like a fragile thing if some of those people leave. If customers depend on it, it might be advisable to switch to standardized tools for managing KVM environments. oVirt is the upstream project to Red Hat Enterprise Virtualization, i.e. those guys who really know KVM.
http://www.ovirt.org/OVirt_3.0_Feature_Guide [ovirt.org]
oVirt has pretty much everything he could ever dream of - and it is well documented, so a
What's wrong with ILO? (Score:1)
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
If it is not broken... (Score:2)
I use serial comms at work frequently... (Score:1)
and I tend to use /usr/bin/minicom. /usr/bin/screen.
Some of the guys at work prefer
Both have huge followings and have been around a long time.
Screen sounds like it may do what you want easiest.
But neither of these are going to work for the power management aspect.
I think you may find you will need to use your custom scripts for that with most solutions.
Cereal (Score:1)