Forgot your password?
typodupeerror
IT

Ask Slashdot: IT Staff Handovers -- How To Take Over From an Outgoing Sys Admin? 195

Posted by Soulskill
from the don't-use-the-words-lame-duck dept.
Solar1ze writes "I've just started a role in an IT services firm. I'm required to take over from an incumbent who has been in the position for three years. What are some of the best practices for knowledge transfer you have used when you've taken over from another IT staff member? How do you digest the thousands of hosts, networks and associated software systems in a week, especially when some documentation exists, but much of it is still in the mind of the former worker?"
This discussion has been archived. No new comments can be posted.

Ask Slashdot: IT Staff Handovers -- How To Take Over From an Outgoing Sys Admin?

Comments Filter:
  • by Anonymous Coward

    Hope to Christ he took good notes.

    • by icebike (68054) on Friday August 02, 2013 @04:08PM (#44460481)

      Hope he is leaving on a good note, and not holding grudges.

      Then systematically go through each machine for which he has a password and have him record these in some secure password vault application of your choice. And also any root passwords he has. Passwords to routers, print-servers, off site corporate backups, corporate accounts (supplier's web sites etc), certificates owned, domain names, email accounts, etc. (You'd be surprised how many small to mid sized businesses wake up two years hence to find their website unreachable because the renewal went to some gone-guy's inbox and/or bounced).

      Go over the system layout (map of the network, interconnects, lans, NAS's, servers, etc), and for EACH NODE, ask if anything has been changed since it was created. If you ask if the document is up to date, he'll just say "pretty much" but if you go over it one router at a time, he will remember things that don't appear in the notes for one reason or another.

      But mostly pray he's leaving happy, and not pissed.

      • by khasim (1285) <brandioch.conner@gmail.com> on Friday August 02, 2013 @04:24PM (#44460643)

        If he is leaving happy, get his contact info and ask if you can check in with him in the future if you have more questions.

        Most of the issues I've run into over the years did not center around HOW something was done but WHY that particular design was chosen. Usually there's one or two weird items at every site that the rest of the system has be designed to accommodate.

        • by thereitis (2355426) on Friday August 02, 2013 @04:48PM (#44460879) Journal
          You might get a couple of freebies with his contact info but I suspect it'd be better policy for an installation this size to set up a paid arrangement with the outgoing sysadmin. I'm not in IT so I don't know what precedents there are around this, but relying on him to reply for free just seems against human nature.
          • by egamma (572162)

            You might get a couple of freebies with his contact info but I suspect it'd be better policy for an installation this size to set up a paid arrangement with the outgoing sysadmin. I'm not in IT so I don't know what precedents there are around this, but relying on him to reply for free just seems against human nature.

            This is great advice. A two-week after-hours contract with the admin for after he leaves is a great investment.

            • by ICLKennyG (899257) on Friday August 02, 2013 @08:01PM (#44462573)
              I would argue that the best way to ensure they leave happy is to pre-pay some token amount of that contract. Nothing ownerous, but say 2 days salary/16 hours up front would be an excellent way to grease the wheels. If you are of sufficient size, the roughly $500-$1000 parting gift is a small price to pay for an enterprise phone-a-friend life-line.

              Hopefully, they are leaving as an advancement not out of recourse. As someone in the incumbent situation right now with evenly mixed feelings, a small olive branch saying "we know we will be at a disadvantage without you and would like to buy 8-24 hours of your time when you have it available over the next 6 months, keep it anyway if we don't," would go an awful long way to helping me answer a phone call or any other question that isn't oh yea the password is 'RumSkittles3242#$@%_god'. That said, without that, if anyone besides my supervisor calls when (not if) the project they are working on fails, I'm going to say "HAHA, told you so!" and hang up.
        • by Spazmania (174582) on Friday August 02, 2013 @05:29PM (#44461301) Homepage

          If he's leaving happy, ask him (and your boss) to work out an hourly consulting rate so you can reach out to him for the next few months and he'll be properly compensated for it.

          • by Common Joe (2807741) on Saturday August 03, 2013 @02:20AM (#44463783) Journal

            I was a programmer for a small firm when I gave my two weeks. I offered them to come by now and again on on Saturdays or answer questions they may have had. Although they didn't call me often and I gladly went over there a few times, I did have to put my foot down and ask them to stop calling me after 6 months. I thought that was enough time for a transition and I only offered my services to be nice... not as a permanent solution to their inability to hire enough people to read and parse my code. The company didn't really want to look at my code or study it or become familiar with it until they needed a change and then they called me up so I could explain things to them. After reflection, I think most companies would either abuse my kind of offer or never call. Would I do it all over again? Yes. I'm a nice guy at heart and I'd make the same offer to the same people. They were a good bunch to work with.

            I put this out here as a tidbit of info for others thinking about doing this.

      • by houghi (78078) on Friday August 02, 2013 @04:50PM (#44460897)

        But mostly pray he's leaving happy, and not pissed.

        That is basically with every person you need to replace.
        If this is an issue at IT level, it will be an issue at every level.

        If it is an issue, then the 'hit by a bus' problem will also exist.

        At every company I have worked, I see it at some level. Only one person responsible for some task. Sales rep who is the only person who has contacts with specific customers. HR person the only one who does salary. Accountant the only one who knows the password to critical files.

        The first thing I try to do when I get at a company is to get away from the 'This is my problem/customer/whatever' and go to 'This is the companies problem/customer/whatever'. This will not be easy for people who feel insecure.

        Try to ask 'what department is responsible for ...' instead of 'who is responsible for ...'. And remember : Graveyards are full of irreplaceable people.

        • by icebike (68054) on Friday August 02, 2013 @05:03PM (#44461053)

          The big difference here is that some filing clerk or HR drone, or Sales exec leaving, pissed or not, does not put your entire infrastructure at risk.
          A pissed sales exec might try and take his customers with him. The HR drone won't be missed, they are a dime a dozen.

          But the Sys Admin, leaving pissed, can put you in a world of hurt by just changing his phone number, not doing any skulduggery.
          A vindictive ex-sysadmin can put your company down for the count months or years in the future, when you least expect it, from a cafe in Puerto Viarta.

        • by neye_eve (212185) *

          "And remember : Graveyards are full of irreplaceable people."

          Man, I wish I had mod points to give today, because that's a pretty awesome quote :-)

      • by Synerg1y (2169962)

        Everything here with the addition of asking why something was done the way it was if you're unfamiliar with it.

      • by Penguinisto (415985) on Friday August 02, 2013 @04:59PM (#44461009) Journal

        Some other bits:

        First, oddball configs - that is, take notes on any custom settings and processes.

        Nothing is more irritating than to troubleshoot something, only to find that the configs are some goofball way-out-of-the-ordinary rigging that somehow works in spite of itself. Or worse, discovering that what looks like a straightforward deal becomes a messy multi-day-outage when you try to fix it according to best practices.

        Sure, you can build-up a replacement that has far better/standard configs, or put together better processes... But that doesn't help you out when $system is down and your users want it back *right now*. It's better to at least get some insight into why it was set up the way it was, and you can then plan of rectifying that before it goes down (and as a bonus, knowing how and why it's rigged like it is, making t-shooting a lot easier to do.)

        Also, I'd get some insight into what projects he had planned and in process - those will give you some insight into what you yourself will really want to pay attention to. For instance, if there's a backup improvement project planned, it may well be because the existing backup solution either sucks balls, fails any integrity checks you may have, or is about to collapse any day.

        Finally, sit the admin down and go over all vendor-supplied services and service contracts (service, certificates, etc), and find out what's about to expire. It would kinda suck if you have your SAN (or worse, core switch, Oracle DB product, etc) crap out, then discover that the platinum 4-hour service contract attached to it expired a week after that guy left... per-hour charges are brutal, parts are moreso, and if your company does the whole PO thing? It's gonna suck.

        Overall though - wring that guy's brain out, and record it to audio if you can. It'll save you a lot of headaches down the road.

        • by cusco (717999) <brian@bixby.gmail@com> on Friday August 02, 2013 @05:42PM (#44461447)
          One more thing that I would add, something non-technical, is ask the outgoing guy who in the organization has caused him the most problems. It might just be the idiot CFO who thinks the Sys Admin is the one that needs to fix his laptop when the latest version of AOL has hosed it, or the branch manager whose answer to every network issue is to yank the power plug on the router to reboot it, but sometimes it can be more troublesome, like the mainframe admins who deliberately try to obstruct projects carried out by the Windows admins. Get a really good handle on the workflow, ticket tracking, and reporting requirements as well (I didn't and am still floundering sometimes).
      • by rickb928 (945187)

        Assuming you both can find or figure out or guess each device's existence.

        Without a current inventory, you are probably doomed. Something will be missed.

        I've left a couple of positions on bad terms, but never, never refused any information. I've inherited several positions, and at least half of them were give to me with the incumbent angry as hell. Anyone remember the Emergency Boot CD? I used EBCDs during the NT era lots of times to hijack the Administrator's account on PDCs and BDCs after hostile dism

      • In other words, you're doomed. If he's working for a mid to large company, and they're giving you only a week to get up to speed...run. Something is terribly wrong.

        Normally, a junior network admin is taken on for several months to years before a network admin moves on. That's to ensure that the entire amount of knowledge is successfully transferred from one individual to the other...and there is a lot to learn in many cases. Some places have multiple network admins, from senior, to mid-level, to junior, wit

    • Hope to Christ he took good notes.

      Trust but verify a list of things.

      • Passwords to systems.
      • Passwords to data bases.
      • Vendor pass words.
      • Backup methods
      • Backup passwords.
      • SSH known host keys
      • Source tree accounts and locations.
      • Local build bits.
      • Environment variables
      • Router and other device passwords.
      • Physical keys.
      • Does the individual have off site backups -- many should and do up to the point that they leave.
      • Terms and fees for consulting...
      • Management IPMI keys and passwords.
      • Domain registry contact changes.

      Change and verify each password

  • Ask to see the last year's quarterly reports that went to Department Supers with signing power over budgets.
  • by Anonymous Coward on Friday August 02, 2013 @03:54PM (#44460315)

    Run like hell....

  • by Anonymous Coward

    Did this recently. Started with core network topology documentation, moved on to DNS. Foundational stuff. Documenting subnets, figuring out what documentation and systems should be deprecated. Made lots of diagrams. Reviewed monitoring tools. Prioritized systems by importance to review for best practices. Got a network security audit to find holes. Bam.

    • by skids (119237) on Friday August 02, 2013 @04:46PM (#44460853) Homepage

      Yep, whenever you change core IT people, you have them do a sloppy braindump if possible before they leave, and you clear the new guy from almost every task save updating the documentation and diagrams, with a few mundane tasks thrown in to get procedures down. This means you postpone your big projects if you have a staff change instead of expecting the new guy to shoulder that. Skillsets are not the same as in-situ knowlege.

      • More than one person should be responsible for the network. (Or in my case, coding a program.) There should be a standardized way of doing things so the new guy doesn't have to come in and hit a brick wall on his initial sprint. Diagrams and documentation should be part of the every day life. I have to admit, though, that every place I've worked at and most places I read about do not have overlapping job responsibilities nor documentation that is worth anything. (I've seen lots of documentation, but th
  • Ideas (Score:5, Insightful)

    by funky49 (182835) on Friday August 02, 2013 @03:58PM (#44460349) Homepage

    Primarily, you'll want to build an honest rapport with the other person. Get inside their head a little and allow them to brag A LOT. Ask how they found the place and what they did to change it. You'll want to breeze through all of the high level and important documentation first so you'll have a baseline. Take as much notes as you can. Ask what websites/resources they use to make it easier to follow in their tracks. Explain your situation to them. It will humanize yourself in their mind and you might be able to engage their compassion for you. Perhaps they would be available to answer questions after they leave! Is there budget money for them to be used as a compensated resource? Hopefully they like the idea of helping others and putting some scratch in their pocket.

    Bon chance!

    Steve

    • by cbelt3 (741637)

      Good approach. Making a mortal enemy of the outgoing sysop, or simply his object of scorn, will screw you badly.

      Other than to say "you're screwed", the big step is to also ramp up the professionalism and start building a better system governance policy and documentation process. The best way to explain that to management is to ask "Do you fail the Hit By a Bus Test? ".... If your key administrators are hit by a bus, will your systems go dark ?

    • by AmiMoJo (196126) *

      In software development we have something called the "code mocking [dilbert.com]". There must be something similar for network admins. As well as gaining an understanding of how the network operates you will also acquire a stockpile of excuses for when things go wrong. Blaming the last guy is industry standard best practice.

  • On my current gig, I got one day...
  • >> How do you digest the thousands of XXXs in a week?

    Dude, step away from SlashDot RIGHT NOW.

  • by schneidafunk (795759) on Friday August 02, 2013 @04:01PM (#44460399)
    I would start by writing your own manuals and have the outgoing person review them.
  • Take over right away. Don't let him do anything. Ask lots and lots of questions. Take notes.

    1. Get da passwords. Verify them.
    2. Support contracts.
    3. "What are common problems"
    4. "Can I get your email" :)

  • by Anonymous Coward on Friday August 02, 2013 @04:03PM (#44460419)

    Make sure you have thick gloves and sanitize everything. Check for booby traps. Never push any buttons till you've traced the wires back to their origins.

    http://bofh.ntk.net/BOFH/

  • Eat his brain. Just be careful of kuru.
  • by sdinfoserv (1793266) on Friday August 02, 2013 @04:04PM (#44460437) Homepage
    1) Need passwords... immediatly change them.Exiting person should have no futher access except through you.
    2) Require exiting person to produce network diagram. Make it their last duty if one doesn't exist.
    3) Now starts the pain... audit devices and systems for rogue accounts.
    4) document as you go.
    5) turn in passwords to supervisor.
    Good Luck
    • by sconeu (64226)

      ha ha ha ha....

      2) Require exiting person to produce network diagram. Make it their last duty if one doesn't exist.

      "What are you going to do? Fire me?"

      • Indeed. The theory is that they will hold some small change over his / her head, like their last paycheck...but it's a gamble, at best (do you really want to piss off the network admin who can, with a few keystrokes, just wipe out your company? or better yet, refuse to help you when a problem, not of his / her own making, does arise? you don't want them pissed off at you, not when they have nothing to lose). Think about it...the company tries to show its strength by asking for some laborious process during

        • by ruir (2709173)
          You are not dealing with MacDonalds employees or exactly cash strapped people to be so naive as to threaten a upper-level employee to withdraw their last check. They will probably send you to hell. Heck, I have refused gigs rather than dealing with crazy people. Plus, if you are asking this kind of information in the last week, you are doing something wrong.
    • 1) Need passwords... immediatly change them.Exiting person should have no futher access except through you.

      Insane. The last thing you need is to lose access to something and spend your entire week talking to support trying to crack a device. Besides, they are probably still finishing up some work. You need to trust them and treat them with respect. On their last day, change passwords to protect the outgoing admin. That way they cannot be blamed for anything after that time.

      2) Require exiting person to produce network diagram. Make it their last duty if one doesn't exist.

      Network? We're talking about a sysadmin. I can't tell you how many times a company has detailed networking doc and monitoring, not no

  • The incumbent will know what to teach you if you only have a week. If they are leaving on good terms, they probably won't be adverse to having some questions asked by email after they leave, so it's not the end of the world after 7 days.

    I've left jobs almost a decade ago and still get an occasional question once or twice a year. It's not that it wasn't documented or couldn't be solved through a few hours of investigation, it's just that a 2 minute email and a 2 minute response later you get your answer and

  • Keep them on as a consultant, and *pay* that $$$ per hour when you need to.

    (This assumes they are quality folk in the first place, of course.)

  • Secret Server (or something like it) is very cool. Get the outgoing person to put all the access passwords, locations, etc. for every bleeping system in it.

    Then change the master password after he leaves.

  • If possible, build a relationship with the outgoing administrator. Accept that a lot of his head knowledge will dissipate soon after he leaves the company but it wouldn't hurt to have him as an information resource for the first few weeks, just don't abuse the privilege. If he's moving on to another job, his loyalties and focus will be to them not his old company.

    Get his permission and comb through his corporate inbox and home directory -- dump it to an offline location. I know you don't need his permiss

  • Now if the old guy worked for the IT services firm then you should not have the outsoureing roundabout in the way.

    But if they worked at the place that the firm took over then there may be paper work / other things that get in the way like some things that the old IT guy used to work on are now not part your IT firm systems or things that you control / are part of your pricing.

    Or other stuff the like the non manged box that does not have uses log to in but runs say the phone switch or the one that runs the d

  • by jellomizer (103300) on Friday August 02, 2013 @04:17PM (#44460579)

    The first thing is to figure out what are the Most Mission Critical systems, and cover them in order of priority, really try to press the criticalcality of the system.

    Top Priority: Systems where there is a Downtime has an immediate impact. There is NO Work Around, it needs to run
    High Priority: Systems where there is downtime work around and they can tolerate it down for a few hours while you mess with it
    Medium Priority: Systems that can be down for a Day
    Low Priority: Systems that can be down longer then a day

    Try to get the passwords, or make sure you have a passwords and rights to all the systems work in order of priorities.
    Create a network map, inventory every system, switch and router... Make sure you have access to them.
    Find the Power Users in the area, they may be able to help you out later on, they may not know everything the sysadmin does but they know their little section and sometimes has tips and tricks that don't get passed on. If there is an issue after he leaves you have contacts.
    Get the vendor support numbers if available.
    Working in order or priority find the custom stuff programs/scripts etc... Do an overview on what they do, what language affect what systems...
    On the second to last day, shadow the old admin, on the Last day do everything, he should only mentor.

    After he leaves. CHANGE ALL THE PASSWORDS he knows, and check for back doors in the network to prevent him from entering the system.

    Due to short time of transition you will probably stumble a bit, but you should have enough to hit the ground running.

     

  • by NikeHerc (694644)
    Make sure the person leaving knows you'll buy him or her a beer or two when you have a question you can't figure out on your own in reasonable time.
  • It is the only way to be sure.

    • by xystren (522982)

      Making two different sci-fi quotes, from two different universes and while still keeping it on topic... Simply brilliant!

      Bows down to your quote-fu

  • Every conversation you have with the outgoing admin, record (with permission, of course). When they're showing you something a workstation, screen capture it. Write notes up for all of this the first while its still fresh. Have them walk you through each server, each device, and all the issue with it. You won't remember everything they've said, and they aren't going to do as great a job documenting things as you'd like during their last week, as they're head's already out the door.
  • When I quit my last job, I was there for 5 weeks after saying, "I'm gonna go ahead and leave." I said I could stay as long as they needed to have a smooth transition but that was clearly a mistake. 2 weeks in, absolutely nothing had been done to transfer my tasks so I set a firm date for 3 weeks later. Had all my tasks documented but no direction on who would take over. Another week goes by. "Who is taking over these tasks?" And another. "Who is taking over these tasks? I would really feel more comf

  • I have been doing this for the last 18 months, since our sys admin was terminated. Write stuff down. Find a secure place (or two) on the network to store an Excel spreadsheet with IP addresses, dns names, and credentials for servers, databases, routers, printers. Encryption keys, vendor support websites. Save root, administrator, and sys passswords, and any other admiinistrivia, in some sort of order you can decipher in 3 months at midnight. I use worksheets to identify categories of information.. It's prob
  • Seriouly, there's no hope you'll actually be able to cram everything you need to know into your brain and make it stick. You need runbooks [wikipedia.org].

    Here's a Technet Article [microsoft.com] on how to put together a Windows server run book. You'll also be able to google for Linux or Unix examples, although you'll find mostly snippets focused on how to write a runbook section for one specific product or another.

    A high-level runbook should document overall systems architecture: network layering, external and important internal connecti

  • Overview the available documentation, talk with the guy if he is available. A lot of times I ended up reverse-engineering everything although so be ready for that possibility.

  • ... for an enterprise critical app at a large corporation (why I wore so many hats is a long story), I was introduced to my replacement with about 60 days to go. I gave him a copy of my 'run book' (you real life admins call them) and showed him where all the requirements and development documentation was. I told him that I'd be redirecting some (easier) administration issues to him and call him in on the tough stuff for the next two months.

    Most of the stuff was 'glued together' with Perl. So one of the jo

  • The short answer is: you don't.

    You fake it until you make it, or can corner the guy in a dark alley and beat it out of him.

    In all seriousness, I've been there probably more than most. I've worked most of my career for managed service providers (of varying quality) where there is no environmental documentation to speak of, in most cases, and almost invariably things are a complete goddamn mess because they're going from in-house to outsourced for a reason. More often than not, you're expected to replace 3-5

  • ... today wasn't your first day at Bluehost.
  • which is under a suitable jurisdiction. The waterboard them until they have told you everything. If somebody asks: terrorists attacked your network.

  • Spend a year documenting the network with the guy present, it's really the only way to do it. Anything else and basically you are screwed, get all the login codes, pray he doesn't forget any and start firing up the network scanners like nmap and logging in and documenting everything.
    • Nmap kills some things like HP jetdirect external boxes, crashes some Samsung office phones and has other consequences so beware of using it on an unknown network during working hours. It's not that there's nothing wrong with nmap, it's just that some network hardware is fragile pieces of shit that will fall over (or in the case of that HP stuff, break) if you hit some ports with packets.
  • How do you digest the thousands of hosts, networks and associated software systems in a week, especially when some documentation exists, but much of it is still in the mind of the former worker?,

    My recommendation is you demand that your boss get the old guy on retainder to be available by phone and e-mail for at least 30 days.

    You really need 14 days worth of acclimation.

    Get as much as you can out of him, bring a notepad and pencil, and a voice recording device.

    I recommend you start with him get

  • On the bright side, you'll improve your forensic skills.

  • It's amazing what holes in the docs show up when things really go wrong. You can pre-empt this by acting as if things has gone wrong and try to find a solution. This should remind the outgoing guy of what is important to pass on.
  • Not an admin but this is what I would consider doing with management backing.

    Install a pair of SSH relay servers with full logging of everything going in/out of it to a write only filesystem. Configure production boxes to only accept connections via this relay server.

    There are good reasons to have a full audit of everything admins do; and suddenly you know absolutely everything the admins are doing today.

    If a ticket is closed and the process isn't documented, give your technical writer the log snippet so th

  • Pay the departing sysadmin for their time, by any legal means, to provide additional information. I've had to work with companies where a core admin had just departed, and had to help hide that we then hired one such admin as part of our company with a different title in another group, partly so we could tap them legally for information about their old company's environments. We got a good engineer, they got a good contract to help out while they looked for a permanent role, and were able to factor in undo

  • ...should be the ultimate goal. Understand the design, get it under basic control and then work with a team (largest you can muster) of diligent specialists to design replacement systems that are firewalled off from the original. The reasons for this are twofold:

    1) No matter how well documented, well designed, etc. the system is, your knowledge of it will never be perfectly complete and you'll never be able to turn around changes with the same degree of confidence and alacrity as the original admin.

    2) Your

  • I already took over systems in both scenarios, friendly and very unfriendly. I agree with another poster the ultimate goal is to reimplement most of the systems yourself. My last takeover was downright hostile. I had to do an audit of the systems, and for instance albeit I had a list of passwords, many were swapped or I had to find passwords for MySQL servers in logs or in scripts. I also found a couple of *very obvious* backdoors. First thing I did when taking over after documenting all systems passwords

Whoever dies with the most toys wins.

Working...