Forgot your password?
typodupeerror
Virtualization IT

Ask Slashdot: Tools For Managing Multiple Serial Console Servers? 104

Posted by timothy
from the many-roads-to-rome dept.
An anonymous reader writes "I've recently been charged with updating our existing serial console access tools. We have 12 racks of servers each with a console server in it (OpenGear, ACS, and a few others). Several of these systems host virtual machines which are also configured to have 'serial' management (KVM, virt serial). In total it comes to about 600 'systems.' All the systems also have remote power management (various vendors). Right now our team has a set of home grown scripts and a cobbled together database for keeping this all together. Today any admin can simply ssh into the master, run 'manage hostname console' and automatically get a serial console or run 'manage hostname power off' to cut the power to a system. I'd rather use some tools with more of a community than just the 4 of us. What tool(s) should I move my group onto for remote serial/power management?"
This discussion has been archived. No new comments can be posted.

Ask Slashdot: Tools For Managing Multiple Serial Console Servers?

Comments Filter:
  • by Havokmon (89874) <rick@havo k m on.com> on Thursday November 07, 2013 @02:45PM (#45358887) Homepage Journal
    If you published your tools, you might find you're not the only ones who need the flexibility you've written into your toolset.
    • by swalve (1980968)
      What does "a community" have to do with whether the tools work or not?
      • by Havokmon (89874)

        What does "a community" have to do with whether the tools work or not?

        To Quote - " I'd rather use some tools with more of a community than just the 4 of us."
        He also never said that there were shortcomings in the toolset they created. It sounds like he may not like the database, maybe he wants a nicer front-end for managing the tables? But it's not described as 'the problem'.

        Therefore, if they create a community around their own toolset, then the only problem actually described in the OP is resolved.

    • HEY! HEY! We want flexible tools, not a bunch of malware! Consider hosting your own FTP?
    • Well, maybe not SourceForge [slashdot.org] specifically, but yeah.

  • ... like OpenStack. Then you have access to everything.

  • Why? (Score:5, Insightful)

    by Anonymous Coward on Thursday November 07, 2013 @02:48PM (#45358937)

    You haven't identified any missing features or existing anti-features in your current toolset.
    The only hint of anything wrong with your setup is "I'd rather use some tools with more of a community than just the 4 of us."

    Q: What tool(s) should I move my group onto for remote serial/power management?
    A: Yours. Clean the tools up, open source them, and market them. Your community will grow.

  • by Anonymous Coward

    You might look at Digi.

    I've used their stuff and it works well.

  • Conman and Powerman (Score:5, Informative)

    by decepetion (632646) on Thursday November 07, 2013 @02:51PM (#45358973)
    • by trandles (135223)

      Seconded. I use these every day.

      • by Troy Baer (1395)

        Thirded. Conman & powerman rock.

        • by Pedahzur (125926)

          Fourthed; again, use almost every day. Conman and powerman are great.

          Conman: Serial consoles, IMPI serial consoles, write your own wrapper if one doesn't exist.
          Powerman: Can control just about everything out there. Again, write your own if it doesn't exist. Can even "manage" power on a VM host. You might have to write your own wrapper though (basically a remote shell that runs virsh commands). I know we have a script here, but I don't think I'm allowed to put it in public. It uses rsh to connect to the VM

  • by Anonymous Coward on Thursday November 07, 2013 @02:51PM (#45358977)

    I would recommend a look at http://conserver.com/

    • by TheCarp (96830) <sjc&carpanet,net> on Thursday November 07, 2013 @03:16PM (#45359229) Homepage

      I actually was about to post that....with one caveat: Look carefully at versions

      Admittedly I haven't worked with conserver since 2005, but it was solid then and I can't imagine it has changed much since. The last time I grabbed it, I found that the source had forked a couple of times and the name "conserver" actually refered to 2 or three different versions of the same program, each with slightly different feature sets.

      Few things are more frustrating than trying to figure out why a program isn't working because you are reading the docs for the wrong one.

      That said, the conserver we ended up using was simple, lightweight, and did exactly what we wanted.... provide named console access from a single place.

    • by tbuskey (135499)

      Conserver is great. I've used it to monitor Linux consoles after boot (via grub handoff to serial console). Serial consoles are cheaper per port then KVM port and you have a log you can grep.

      I've also used it to monitor several consoles going to embedded devices. The users could take over when a coworker had gone on vacation w/o calling the sysadmin.

  • Conserver (Score:4, Informative)

    by Anonymous Coward on Thursday November 07, 2013 @02:51PM (#45358981)

    At a previous shop i worked we ran conserver against about 1k machines, some virtual but the big bulk of them was physical.

    I also implemented a standardized way to control power on/off/reset on those machines as a patch to conserver. Those patches can be found at https://github.com/glance-/conserver/

  • My company builds a replacement (http://uplogix.com), smarter version of a serial console. They can all be managed by web UI and you can term directly into each device, keep configuration on them, and keep each device mapped to its outlet on the power controller. We even have a virtual version that runs on vSphere. You can hook up all the ports via telnet and keep your existing term server, but getting the benefits of the rich CLI and web UI.

    Sounds like a perfect use case for you.

    • by h4rr4r (612664)

      WebUI? WHY IN THE NAME OF THE FSM would any sysadmin want that?

      People who want serial console access are not looking for "teh shiny".

      • A web UI means I can handle an issue from any device - a computer in a hotel, grandma's cell phone, whatever.

        Also, with many systems, a UI to navigate groups of systems can be handy.

        • by h4rr4r (612664)

          Just use SSH. You can even keep a copy of a client for most OS with you.

          • I see you didn't bother to read one word of my post before replying.
            Does the computer in the hotel business center have ssh installed? No. Does a borrowed phone have ssh? No.

            Ssh is great, until I run out of battery.

            • by h4rr4r (612664)

              A thumb drive does not need batteries.
              You can use an ssh client without installing it from a thumbdrive.

              You can install a free ssh client on that phone.

              When I am on call I always carry my charged external battery pack.

              • Try running some random executable from a thumb drive on a hotel computer and tell me how will that works.

                Then the next person you see, ask them if it's okay for you to install weird "hacker" apps on their phone

                • by h4rr4r (612664)

                  I do not work from either of those. Do you really trust neither have key loggers installed?

              • by muridae (966931)
                A thumb drive does not stop the hotel computer from having a key logger, unless you check the cables and boot into a secure OS. I spotted a physical one attached while traveling (New Jersey, Pennsylvania? somewhere up the east seaboard between Boston and DC). Was hard trying to stop my family from making a hotel reservation over a hotel's computer (that one was booked, trying to reserve a room 50 miles down the road) without screaming "because the computer is bloody compromised and that little box could be
            • by muridae (966931)

              So, wait, you replace SSH on a known secured computer (at least I hope an admin's travel computer is relatively secured) with a web UI? So you can use it from a adware, spyware filled device like a hotel lobby computer or grandma's cell phone filled with spying, keylogging games? Sure, the web UI might be over HTTPS, but that does nothing about spyware seeing you punch in the URL, then type in your username and password. I really hope your IDS knows when you are traveling, and will use the website, and when

              • That's a good point and I retract my comment in the context of console servers.

                The point I had in mind is that although I use CLI for almost everything, sometimes a GUI is much nicer. The CLI for LSI RAID cards comes to mind.

                • by muridae (966931)

                  Locally, sure. Heck, dropping a UI in front of the scripts that the article was asking about might make the management of the variety of devices easier; Just toss together a python program, or other easy language at hand, with the ability to call bash scrips and the ability to throw up a UI and mess with the database that tracks all the devices and their login details; that would make their scripts more usable should they all get fired tomorrow and the new team has access to an easier work flow.
                  But remote a

            • by tbuskey (135499)

              If you're using the computer in the hotel business center to get console access, you probably don't care about security. If you care about security at all, you're going to use your own device.

        • by Sique (173459)
          Web UIs suck if you have to mass deploy something. CLIs are predestined for such jobs.
      • by tthomas48 (180798)

        The advanced CLI is for the sysadmin. The WebUI allows you to lock down users to say 1 port on a machine and give them a nice shiny button to click on.

  • by tudorxpopescu (2595289) on Thursday November 07, 2013 @02:55PM (#45359023)
    So... let me get this straight, you have a system which is easy to use and works just fine, and is written in house. Obviously, you want to change it... because? Jeez.
    • by rnturn (11092) on Thursday November 07, 2013 @03:09PM (#45359157)

      My thought exactly. Unless they expect their console access needs to explode soon and the current system cannot scale, I can't see the need to change. The existing crew knows how to use the current setup and surely no more than a couple of pages of documentation would be all that's needed for any newcomer to come up to speed. Switching to a 3rd-party console access tool will just be one more thing that'll wind up appearing on the job adverts for new administrators and one more thing that'll slow down the hiring process when the HR filtering software doesn't see `ConsolePro2013++ Gold' on a candidate's resume.

      Or did Management issue an edict of "Buy Not Build"?

      • My thought exactly. Unless they expect their console access needs to explode soon and the current system cannot scale,

        Agreed. Standardization should decrease complexity (of maintenance), not increase it. Keep the tools as simple as possible, and as close to vendor standard as possible. It's like admins who customize root. If you leave it generic, then every admin knows what to expect.

        • It's like admins who customize root. If you leave it generic, then every admin knows what to expect.

          ... and every intruder too!

          • by bobbied (2522392)

            Now *how* would an intruder get root?

            Sarcasm, aside...

            There are benefits to keeping things standard and as long as you implement reasonable security configurations and don't allow non-local root logins, I don't see any real reason to change root in most cases. Your life (and the lives of admins to come behind you) will be much easier.

          • by TheCarp (96830)

            I completely agree except for one thing.... and I am looking at YOU redhat..... if the system default that they come to expect is that ls output is unreadable due to dircolors being the distribution default which assumes a light colored background.... that should be fixed with extreme prejudice.

      • by mu51c10rd (187182)

        when the HR filtering software doesn't see `ConsolePro2013++ Gold' on a candidate's resume.

        Hmmm...and I am only experienced with ConsolePro2011++ Gold...time to upgrade the skillset...

    • Because virtual server cloud crap is all about resume buzzwords, brah.

      It doesn't matter if you've produced an excellent in-house solution, because THAT DOESN'T GET YOU A JOB ELSEWHERE. Much better to switch to something which'll end up with your spending 50% of the original time adapting to the new system, and another 80% implementing all the missing features.

    • by sl4shd0rk (755837)

      Obviously, you want to change it... because? Jeez.

      I think OP is trying to address the issue of having an infrastructure running on a bunch of home grown stuff supported by just a couple guys. If it works, great but when one or two or three of those guys leave, all of a sudden nobody knows how the thing works and you have a mess. If there is a mandated documentation ritual along with revision control, it helps, but that's probably not the case. OP is wondering if there is an open source solution out there already doing what they've duct-taped together. In

    • by Gothmolly (148874)

      No kidding. Large enterprises are built from cobbled together scripts and databases.

  • by tlambert (566799) on Thursday November 07, 2013 @02:56PM (#45359029)

    Are you willing to monoculture your vendors?

    If the answer is "no", then you are stuck with your home-grown stuff. Vendors intentionally introduce incompatibilities to lock you into using only them, so you aren't going to find some project that provides a HAL, or at least not one that will live through the next software update from one of your vendors.

    You should also be aware (I'm sure you are, if you understand the dynamics of your scripts, but some reading this probably aren't) that some systems won't negotiate a KVM style console unless they are selected active in the KVM prior to boot, so there's an interaction between your power management sequencing and the virtual serial and real serial, and that varies from vendor to vendor and software update to software update.

    If you are also using Real KVM(tm) style virtual video consoles, you're probably already aware that Linux and most other Open Source OS's fail to negotiate EDID information correctly, unless you use the closed source video drivers, unless they are selected as the active input on the virtual/real video display device, since those implementations are usually not multithreaded, and so if you have 4 HDMI inputs, and #2 is selected, and #4 is where the device is that comes up and does it's one-time negotiation (this is what's broken about the OS drivers: they should retry periodically until they get a response, then echo up the response to the video driver, which if it's in X/Wayland in user space, it's not going to happen, since it only happens at startup) you are SOL. So your scripts probably have that knowledge, too. Not that it'll do you any good if you have a Linux box on the second input on a physical Samsung monitor, mind you, as they automatically switch away from inactive inputs, and default tho the first input if there are none active.

    So good luck with your ask, unless you are willing to start your own project, and are willing to push to get UEFI, u-boot, Linux/BSD video drivers, and other things fixed as part of the project overhead.

    • by h4rr4r (612664)

      WTF are you talking about?

      There are plenty of open source tools that can support lots of vendors. Conman is just the first one I thought of.

      By KVM he means Kernel Virtual Machine not Keyboard Video Mouse.

  • by Above (100351) on Thursday November 07, 2013 @03:00PM (#45359081)

    There's a little known, but very useful program called rtty. You can find it at ftp://ftp.isc.org/isc/rtty/rtty-4.0.shar.gz. Yes, it was last updated in 2003. Yes, there are package for major open source distributions.

    Here's serial consoles on the cheap:

    Buy multiport USB to Serial devices. They are a USB hub with a bunch of USB to Serial adapters hung off of them. Here's a 16-porter for an example: http://www.startech.com/Cards-Adapters/Serial-Cards-Adapters/~ICUSB23216F

    Hang them off a low end box, I like half-depth Intel Atom servers with lots of USB ports.

    Run rtty. It records each console to a log file 24x7, and allows multiple people to connect at the same time (including typing).

    • by bobbied (2522392)

      Problem with this is that every time you plug in a new USB device, you are likely to totally hose the device names on the next reboot. It is *really* hard to tell what order your USB devices will be initialized and in this case it will be a serious issue trying to sort out what port is which every time you want to add or remove a device.

      • by profplump (309017)

        It's pretty easy to find USBSerial interfaces that include a unique ID of some sort. That plus 4 seconds editing your udev config will give you stable device naming.

        • by bobbied (2522392)

          As I understood this suggestion, one was to go pickup a pile of identical USB to serial converters and USB hubs and just wire them all up. There will be nothing unique between the devices except how they are connected to the host USB hub and in which order the devices are initialized.

      • by Above (100351)

        I've not seen a hardware/software combination in the past ~5 years that varied the initialization order. Now, the order in which the USB ports are initialized is often non-intuitive and non-documented, but just plug a flash drive into each available port and reboot the server to find the order they are initialized in.

        Where I see most people go wrong is they don't figure out the order the ports are initialized, and thus don't plug the serial device into the first to be initialized. By plugging into some ot

        • by bobbied (2522392)

          Which makes my point. I'm not saying that the order is not set, but that anytime you make a change to the hardware configuration by plugging something in or removing a device you may unknowingly move a whole bunch of stuff in a way you don't understand. This might force you to have to go back though 120+ devices per host port and sort out what port is which device, just because something changed.

          Consider what happens if one of your first 50 devices fails (say it comes unplugged or the cable breaks). What

          • by Above (100351)

            I think you misunderstood what I said.

            The OS (at least, Linux and FreeBSD) enumerate the motherboard ports in order. So if your motherboard has 8 ports you can discover the order and number them 1-8.

            If you plug in a device to port 1, and then later plug the next device into port 2, there is no chance the first device will be renumbered.

            However, if you plug into port 2, and later add a device on port 1, it will get renumbered.

            So when I get a new motherboard, I take a pile of old 16M (yes, Meg) flash drives

      • by drinkypoo (153816)

        Problem with this is that every time you plug in a new USB device, you are likely to totally hose the device names on the next reboot.

        Look at /dev/bus/usb. Unless you lose a USB controller those device names should be stable.

  • The most important thing is that it works reasonably well and doesn't require excessive administrative overhead. You've described a reasonably well managed configuration, so it's tough to justify changing everything "just because".

    Exactly what would you accomplish by switching to an "open" technology? Answer that, and then you can make the best decision.

  • to manage their servers including console access via a special NIC port and the ability to push the power button remotely through a web browser

    basic features are free. remote console is a $400 per server option called iLO Advanced on HP servers. the management software is free as well with some features licensed per server

    • by drinkypoo (153816)

      Beware the evils of OOB management. Much of the time, that functionality in a PC server is handled by an IPMI module, which is a little computer in its own right which will probably never receive adequate security patches and which has access to twiddle important configuration values in your system. They're wicked cool when they're not being used to own your network, though.

  • "We have something that works and I want to break it"

  • Open the source of your tools and make them available to the public. You will be able to grow your own community. I would LOVE to have a look at your home-grown stuff. We are an Avocent shop here when it comes to remote console/KVM/power access and so far the only tool we are currently looking into is DSView which a) is sucky enterprise-y software, b) extremely pricey, and c) does way more than we need it to do. Thanks to your posting, though, I found interesting links to tools like conman and powerman in t
  • At $WORK we have used Livingston PortMasters to deal with our serial console issues. Network connected and logins can be setup to only have access to certain ports. Have worked well for us. Usually found pretty cheap on eBay.

    • by Nonesuch (90847)
      We did the same, but isolated all portmasters on a standalone switch, as the product line has been dead for years and the security in the product was pretty minimal even when Livingston and then Lucent was actually supporting the Portmaster. I think this guy is more asking about the software to handle the connections/auth/logging/etc rather than asking about a hardware solution?
      • by jlockard (140979)

        Actual implementation for us is also SSH'ing to a *NIX box, then tipping over to the PortMaster.

        • by Nonesuch (90847)
          Sounds very similar to what we developed. Parallel evolution? Or just creative re-use of obsolete Portmasters?

          When we got more powerful hardware for the SSH bastion host, I wrote a set of daemons and scripts which would maintain one 'screen' session for each console port. At startup it would enable logging, make a 'telnet' connection, and then disconnect the session and leave it idle. When a user wanted to access a port, they'd run a menu tool (setuid launcher and Perl, iirc) that would give them a

  • Not sure if this is what you mean, but we use an MRV server that has a bunch of serial ports and provides an SSH port corresponding to each serial port.
  • If the current system works well then don't try to fix it. " I'd rather use some tools with more of a community than just the 4 of us." is a very bad reason to try to install and learn a completely different system.

    Unless more information is presented, I'll place this in the "idle hands are the devil's playground" category.

    • by LesFerg (452838)

      Well one aspect to consider is, keeping a bunch of staff tied into using some proprietary internally developed tools could also be isolating those staff from gaining any experience with current tools out there in the market and in use by many/most future employers. I am stuck in a situation like this myself, where any chance to get a new job will rely on me learning new development tools in my own time on my home PC, then trying to convince somebody that this is the same as workplace experience with the to

      • So you are suggesting that the employer should risk messing up a working system so that the current employees could develop a skill they could use at a new employer?
  • Disclaimer: I work for these guys: http://www.ovirt.org/Features/DRBD [ovirt.org]

    As somebody said before, this shop sounds like a fragile thing if some of those people leave. If customers depend on it, it might be advisable to switch to standardized tools for managing KVM environments. oVirt is the upstream project to Red Hat Enterprise Virtualization, i.e. those guys who really know KVM.

    http://www.ovirt.org/OVirt_3.0_Feature_Guide [ovirt.org]

    oVirt has pretty much everything he could ever dream of - and it is well documented, so a

  • Im a little lost.. why not using ILO/ILOM/DRAC etc.. its all the same thing. Just give each ILO IP a DNS address and ssh to it, log in and there you have all the tools you need, a way to access the system console, power on/off remotely, check hardware events, hell it can even send SNMP traps to a NMS so you get alerts even if the OS dies! Using serial is so old hat, and clunky. Why on earth would anyone be using it at all these days? Serial! LMAO *scoffs*
    • How long have you got? ILO often hangs (as in the ILO *server*) when extremely long lines are sent to it. Some features as licensed on per-host basis. Unless you only have one supplier you end up with having to use a mixture of tools to do the same operations. I haven't used one of these proprietary tools that doesn't suck big-time. The worst offender is Dell's Java KVM. I have to fire up a Windows VM whenever I have to use it. Once configured serial just works.
      • I usually just use the SSH interface to the ILO and drop to a console from there. Its the same as serial, but with the added advantage of not being serial (9600bps sucks!) Of course you could use IPMI/SNMP/WBEM and code up a tool if you want to be more efficient, but realistically how often do you need to do this? ILOs usually provide good event logging capabilities as well so you can see what happened, when.
        • Debugging installation failures on hosts in a dev environment, probably 20 times a day. I just need to see what the installer reported and to interact with it. These fancy interfaces give you all the bells and whistles but seem to end up making the basics unreliable. 19200 works fine for me.
  • If it is not broken, do not fix it. Your installation seems to work, why do you want to waste time "fixing" it?
  • and I tend to use /usr/bin/minicom.
    Some of the guys at work prefer /usr/bin/screen.

    Both have huge followings and have been around a long time.
    Screen sounds like it may do what you want easiest.

    But neither of these are going to work for the power management aspect.
    I think you may find you will need to use your custom scripts for that with most solutions.

  • Hey there, We are in the process of migrating away from proprietary serial console servers to one that's based on simple hardware and open source software. See http://cmrg.fifthhorseman.net/wiki/cereal [fifthhorseman.net] it's also packaged in debian already.

"The value of marriage is not that adults produce children, but that children produce adults." -- Peter De Vries

Working...