How Do You Interview A Sysadmin Candidate?

benedict writes: "The article No Shortage of Programmers? sparked a really interesting thread about how to interview programmers. Being a systems administrator, I am curious about the Slashdot community's collective wisdom on how to interview sysadmins. I have come up with a few questions of my own to prime the pump. 'What is tcpdump? What is it good for?' 'How about truss/ktrace/strace? What are they good for?' 'What's the largest number of machines you've maintained? What have you done to make it easier on yourself (e.g. what types of automation, file distribution, etc.)' 'Do you use source code control? What for?' I would also present a couple of 'hypothetical' situations from my own experience and ask how people would approach them. How about you: what kinds of questions would you ask, what situations would you describe, what kinds of answers would you look for?"

Three top questions:

Have you ever played Core Wars?

Which weapon on Counter Strike do you prefer?

What is your home machine?

Ask them if they..

Know what BOFH stands for :)

Here's a working link...

...to the No Shortage of Programmers [slashdot.org] article.

May I have your advice...

on this file I send you?

Troll Them

When I'm interviewing someone with whom I'm going to be working with, I'm just as concerned about how well I'm going to get along with them (and how they will interact with the rest of the team) as with their technical experience/ability.

My favorite means of testing this out? Troll them. Ask them which editor they use. Ask them which O'Reilly books they own. Ask them which distro of Linux they prefer. If they're zealots about things like that, its going to make it a lot harder for me to get along with them.

Of course, it's very important that you don't make them feel like they're being grilled/trolled, because they're also interviewing you, and deciding whether or not they want to be a part of your team.

--Cycon

Network Engineer - Hopefully Helpful

I'm a network engineer (Cisco dude) not a programmer, but maybe this idea will be able to help.

I found a really good way to do an interview was to point the vict^H^H^H^H candidate at a dry erase board, hand them a marker, and tell them to draw up the network they most enjoyed working on.

It allows them to take control and talk about what they know, giving them a comfort zone. I can ask whatever questions I think might be useful. I can add or remove a component and find out how they would work around it. I can also make sure that they are comfortable thinking in the same mindset that I have. I can make sure they are talking the same language that I am talking.

How to do this in a programming arena? Instead of a network diagram, maybe a flowchart for the logic, maybe a screen drawing for layouts, maybe pseudocode or code, although I would expect that last one would get hairy on a dry erase board.

Anyone who can build on this, please do so. I got a CS degree doing programming, but that was way too many years (and beers) ago. I do not remember enough to really be useful on this.

Don't give them a pop quiz.

Don't ask them little trivia questions like "What does this utility do?". A sysadmin is more than just a person who does routine tasks every day, they have to solve new problems as they arise. You should interview a person by asking them problems and how they would solve them. Ask them about big problems they encountered in the past and their solution to them, or problems they weren't able to solve.

What you want to find is someone who is interested in what they do and can learn new things as new problems arise. You don't want someone who just memorized a book and some man pages, because what will they do if something happens that wasn't in the book they read?...

Interviewing sysadmin candiates [unix only]

I've interviewed quite a few, and there are 'sets' of questions you can ask. You don't want to follow a work book, but there are a bunch of formulaic questions you might ask:

• Describe the process in which your favorite version of comes boots up from a cold state. Please use as much detail as possible. The advantage of this question is that there are TONS of sideroads to check. Also, you find out how interested in the underlying part they are. Also, you can see what run control scripts they hit, and you can hit those applications later... Or better yet, they can tell you things like what run level 4 on Solaris is, etc. (ie, trick questions)
• OSI layers? BSD vs. SysV? This tests if they are well rounded. You see if they've touched networking, you can see if they even know the book learning on the different OS's, and get general 'you need to read a book to know this' type stuff. Also, asking the differences between things in the simplest possible terms is another good test to see if the candiate has the ability to talk to managers. `8r)
• Favorite OS and why. Any good unix candiate belives in 'the right tool for the job'. Anyone who says that 'Linux is the answer to everything' is fooling themselves. All the different Unix OS's have their advantages, and the key to having them explain theirs. They don't have to agree with you! that's the key. But they should at least make sense. But don't hold it against them if the answer is 'Because I know that OS the best'. It's a common one. But do NOT let them just say 'Oh, AIX sucks' etc. If they can't back that statement up with facts, they obviously havn't looked at it close enough.
• How would you rate yourself on DNS? Ah, an expert, eh? What are the different types of records? What are some limitations of MX ones? Get deep into at least one major unix process. Sendmail, NFS, NIS, and file systems are all very good parts to go into detail on. By asking how they rate themselves, they show either a) they know what they're talking about and rated themselves appropriately and b) They are rating themselves guru-level when they have trouble remembering even the names of the parts of the program.
• So I had this really hard problem... I was seeing this kind of behavior... What sort of things would you check to solve the problem? No, I tried using This calls into all their troubleshooting skills. You see how deep they go, what they go to next, and why. There are a couple sendmail and NFS problems that can run the gamut.

One more thing... Don't expect excrutiating detail on a process that you don't know either. IE, don't ask a person questions that you don't know the answer to either! And if you do feel inclined to ask about something you don't know, make it clear that you're coming at it from a newbie's point of view.

If people have further questions, i'd be happy to answer them.

Destiled wisdom

There is loads of highly concentrated wisdom in here [ntk.net]

Broad Knowledge/Fast Learner

Look for people with a broad knowledge of many technologies, even if they aren't experts. You're looking for people who might not know everything, but have a large enough framework knowledge and the willingness to learn anything new.

The easist way to weed out the "Quick Study Course" MCSEs is to ask them about thier experiences/knowledge on Unix/Linux (even if they don't activily use Linux any competent sysadmin has read about it).

If the position is going to be inside a team, and the interviewee seems pretty comfortable, declare the interview over with. Then take the interviewee to the breakroom/lunch and arraige for the other team members to drift over. (Don't go to someone's office to say hello.. this puts the interviewee on unfamilar turf) Maybe have one of the team members toss out a problem they're working on or give a status report. See if you can get the interviewee to interject ideas or solutions. They'll be pretty nervous, so don't hold it against them for being quiet but the really good ones will love talking shop and may even give some free advice. Plus this makes the team members feel more involved in the process.

The biggest thing I can say.. is look for experiences outside the workplace. If someone did something for fun, odds are good that they learned more about it than they ever could of on the job or in a classroom.

Signal-Words

If they say any of the following:

Robust

Synergy

Think-outside-the-box

Current- state

Pro-active

Throw them out of your office.

The questions they asked me...

CIO: "So what do you do?"
ME: "I play a lot of games."
CIO: "Ever make a UT Server behind a firewall?"
ME: "Yeah."
CIO: "Windows or Linux?"
ME: "Both."
CIO: "Go to HR and get a badge."

True story.

My Best (UN*X) SysAdmin Question

Without a doubt, the best (UN*X SysAdmin) interview question I was ever asked was actually quite simple.

The interviewer brought up an xterm on the terminal on his desk, typed `ls /etc`, then asked me to identify every file in the directory.

For added difficulty, they were using a version of UN*X I didn't have experience with (BSDi). The question tested:

• The breadth of my technical experience (how many of those files did I know or not know)
  
• My communication skills (how well did I articulate what the files were for)
  
• How I responded to pressure (there're a LOT of files in /etc, making the question a bit intimidating)
  
• Since it was an unfamiliar UN*X, it forced me to do some analytical thinking and draw on previous experience to make educated guesses (identified as such) as to what the files might relate to based on name and content (he let me `cat` the ones I didn't know).
  
• Lastly, it gave an indication of my overall level of professional/intellectual curiosity, since a lot of those files will never come into play except in extreme situations.
  

Lucky for me, I'm pretty curious by nature and got the job.

1st question:

Who is CowboyNeal?

Re:1st question:

Handwriting test: if their handwriting is anything but completely illegible, don't hire them.

Eye test: if they aren't near-sighted, just say no.

Wrist test: if they don't have carpal tunnel, nix 'em.

Clothing test: if they show up to the job interview wearing a suit, they have no clue.

Jargon file test: Do you know what RTFM means? Can you recite the entire "Story of Mel"?

Caffeine test: If they don't ask for coffee, tea, Coke, or some other form of caffeine several times throughout the interview, forget it.

Slashdot test: What is your slashdot karma? (Don't hire if Karma 25)

Microsoft test: show them a picture of Bill Gates naked. If they don't turn away and run in disgust, don't hire 'em. (NOTE: a good hire will be very difficult to catch)

/dev/null test: What is the true use for /dev/null? If the answer is not 'for redirecting Web proxy logs' forget it.

A key human resources criterion:

Do you like children?
If the candidate answers with anything other than some smartass reply like "Yes, with lemon butter and capers" then reject.

not just technical too

think what kind of mindset and character you want in your sysadmin... patience, calmness in the face of crisis, attention to detail, able to plan. try and ask some questions to get an idea of this.

technical information can be learned reasonably easily but some things cannot be. when i was interviewing people i looked for people who had a genuine interest in IT too and how easily they could pick stuff up, and how well they could manage a number of concurrent tasks.

Random thoughts

Random thoughts from my past, in sysadmin and elsewhere:

• Last programmer I hired was because I saw her reading Webmaster In A Nutshell on a commuter train. "You wouldn't happen to be looking for a job, would you?" Bing-bang-boom.
• I always ask "How do you keep up?" Good answers: "I read Slashdot", "I have a $2K/year book budget", "I'm trying to get a complete collection of O'Reilly books". More importantly is how they reply. If they have to think about it, then they don't see it as a priority.
• Culture in general is key. Any mention of Slashdot or similar fora is almost a requirement.
• Immersion questions are as important as specifics. Someone who crammed through Teach Yourself Perl In 72 Seconds may well know the three basic data types, but only an experienced programmer will know who Larry, Randal and Tom are.
• Ask him to DO something. I've given printouts of suboptimal web pages to the candidate and said "OK, how would you fix this?" (Tell him it's an old version of the page, so he doesn't have to worry about offending anyone with his comments) Read Nick Corcodilos' excellent Ask The Headhunter for more about this sort of interviewing.
• It's critical that the person actually care and/or be interested in the business that you're in. My company is in the education business, and in my eyes, it's important that they see the industry itself as valuable.
• Ask about what kind of machine they have at home.
• DON'T ask the standard bullshit questions, like "Where are you going to be in five years?", "What are your greatest strengths?", etc etc unless you actually CARE about those things. Chances are, you don't, and you can't even answer those yourself.

Have you ever

Have you ever worked with a mixed platform network? Done a network backup on such a network? (I only mention this because I'm a big fan of backups. Well, I am now that I've found that the hassle has a payoff. A large, large payoff.)

Andrew

I always include 2 questions

1) Tell me about the things that you have done at your last job or two. Tell me about how you got 10 different types of boxes to work together or implemented a system to do something or whatever. I don't really care if you know all the options to tcpdump by heart, you can look them up when you need them. I would much rather know why you have done the things you have done and how.

And 2) Do you have any questions about working here?

Picking a sysadmin

To be honest, I wouldn't ask too many questions on specific commands - that's what man pages are for, and they might have been focusing on other things.

One of the big things I would check for is troubleshooting skills. And in a non-obvious way, so they don't zero in on what you're asking for and give the "right" answers. Asking to give an example of a problem in the past and what they did, give some hypothetical situations (though some people think better in front of a keyboard then in when speaking.)

On big one with me is automation and tools. I don't care if you know a specific tool - that can always be learned. But once you get to real sizes, you need to use automation and tools, you can't do everything by hand. If you told me that you speced out or even wrote tools to fit the specific circumstances of the last job, that's a big plus. Along the same lines, any sysadmin that can't take the time to be fluent in a shell probably isn't worth my time. I ask them for their prefered shell and why. It doesn't really matter what they answer, as long as they have an answer. Along those lines, tellign me that "they used to love [insert shell], but now they don't care as much because they always use perl (or other appropriate language)" is also fine.

Sysadmining is sometimes periods of boredom followed by periods of extreme need. If you can keep your cool in that extrene need, that's very good, but hard to judge on an interview. It's very important, though. If you're a self-starter, and those periods of boredom will be used on projects to make your job easier, either from a manager or self-starting, is also good, and something that might be easier to detect in an interview.

Many sysadmins have a large (and fairly well-deserved) ego. This is almost a "necessary evil". However, a prima-donna or someone who will not work with other team members is a problem, and that can be determined to a point during an interview. Also watch out for loose cannons. They can be great, but they're hard to control. A small company might benefit more then a large one by a loose cannon, but no matter how good they are they can get you in trouble. You just need to balance if it's worth it.

=Blue(23)

Possible Question

If your company is running NT or 2000 servers you might ask "How quickly can you reboot a Windows NT Server?"

Types of question

Interesting question. Here are several points to consider that I think are important.

It's important when formulating the questions for a sysadmin to avoid trying to ask "catch out" questions, and better to have a good stock of "standard" questions that will ensure you know the candidate has a solid understanding of the principles. Knowing all of the flags to "ls" or "tcpdump" for example, doesn't tell you much, but knowing that they understand the differences between RAID 1 and RAID 5 is. Crank up the difficulty as appropriate for the position.

Asking questions that only catch out the candidate, leaves them feeling bad throughout the interview, and you with little more knowledge than what they don't know, and maybe a pointless feeling that you caught them out. If that floats-your-boat, go for it, but not me. been there, done that, thinkgeek ain't got the t-shirt.

Also, once you've identified that the candidate has a good foundation of knowledge, start asking about approaches they've taken to problems. One of my favourite questions is "What's your biggest f#&* up". Everybody makes mistakes. If a candidate can't think of a big fubar situation that they've been involved in, chances are they're either very good or inexperienced. It's also a good talking point to base additional questions around. Bring in your own situations as a way of lightening the questioning. You can reverse the question for the age-old fav "Tell me about your biggest achievement", but I prefer problem solving skills in an SA.

I'm also a big believer in "fit". If the candidate "feels" right, but has made a few boo-boo's in the answers given to questions, better to take them than somebody who doesn't "feel" right, and got all the questions right.

At the end of the day, it's a judgement call, and there are plenty of other factors to take into consideration that i've not mentioned here (and I'm sure others will). In a nutshell, find a questioning style/interview technique that ensures the candidate is at ease, feels they can be honest, and covers all of the main points.

Oh, and personally I hate giving and doing technical tests where they're left to fend for themselves for an hour in an empty office. Wasted time all round. Get somebody to interview them in that time who can get more out of them.

Needless to say, get different people to interview as well. Technical skills are but one part of a good employee. HR departments sometimes come out with very good points all the techies in the world couldn't find out.

Hope that helps.

Re:Types of question

People skills! Programmers can hide in their
caves and snarl when thrown their ration of caffeine and carbohydrates. Sysadmins gotta be able to smile while being asked: 1) to do the impossible, 2) Yet Another FAQ, 3) add another user, 4) restore from backup, 5) say no to some bigwig who wants the security policy violated for his/her personal whim, *) you get the idea.

Be good to your sysadmin, even if it isn't his/her day.

One question I wish I could ask...

...Is for references on the employees he/she has helped.

Lets face it, there are two types of sysadmins:
1.) The type that sits in a locked server room never to be bothered (see BOFH).
2.) The type that wants to help you in a kind manner.

Sure, it is more important to have a knowledgable sysadmin that can knows a ton, and knows some clever little techniques to make everyone's life easier, but its also important to have one that is good with employees and treats everyone well.

One of my former employers had a sysadmin that everyone was afraid to go to because of the tone he'd use. He always shouted and was just generally mean to everyone. He was fired, and the man that replaced him knew just as much, but was always helping people with a smile and would stick with you until the problem was solved. It was a huge difference. People loved the new guy.

I'm digressing, but the point is, a sysadmin job usually requires that you help fellow employees, and that is something to check for in an interview.

Not "what it does", ask "how to do"

Don't ask "what is tcpdump?".

Instead, ask "what would you use to view the contents of TCP packets on the network?"

We start with the basics "what would you use to list the contents of a directory?" and work up from there, to gauge the level of knowledge.

Also, technical folks conduct that part of the interview over the phone, and the person doesn't get a face-to-face with a manager about non-technical issues until AFTER we've made our recommendations.

"What's the most difficult thing you've seen?"

One of the best I've seen and heard of is asking "What's the most difficult programming problem or task you've encountered? How did you solve it?"

It's a good question, because it lets you gauge what the applicant is good at, what they might be weak at, and allows you to see evidence of their ability to learn new things.

In other words, was what they consider "difficult" something you'd also consider difficult? Were they able to come up with an elegant and clever solution? A good duct-tape-and-baling-wire workaround? Were they just plain stumped, but understood a good solution when they saw it? Or were they lost completely?

How to Interview a Sysadmin?

Ask him something he obviously doesn't know the answer to, something he hasn't put on his resume. If he gives you a bullshit answer, kick him out the door. If he says, "I don't know", ask him how he'd find out and listen to what he says. Not every sysadmin knows everything, but the truelly good ones will know how to find the information they need. A sysadmin who says, "I dont know we need to hire a consultant" is not someone you eant working for you.

Re:How to Interview a Sysadmin?

> Ask him something he obviously doesn't know the answer to, something he hasn't put on his resume. If he gives you a bullshit answer, kick him out the door.

Not so fast, marketing is hiring too!

Re:How to Interview a Sysadmin?

No, no! You know the drill! If there's any chance that he'll end up as a marketeer we have to drive a stake through his heart, decapitate him and bury the remains at a crossroads at midnight.

There may be something involving salt as well, but maybe I'm just I'm just thinking about tequila.

Re:Ask open ended questions

"I tend to ask a lot of open ended questions to get the candidates talking. One of my favorites (lately) is "A new peice of software is placed on your desk. How would you go about implementing it into production?""

I always ask a potential applicant to spell "piece" for me.

Read the hobbies. No, really!

A few jobs back I ended up doing a lot of the sysadmin work for a good part of the company. As a joke, my boss dropped a "sysadmin" magazine on my desk one day. It turned out there was a timely article in it... The article talked about how well programming tests work in a dev organization. Some folks did a study where they applied a how _bunch_ of tests to candidates, but did not _do_ anything with the results; they just kept the data around. Two years later they looked at peer feed back, performance reviews, etc, etc, to look for "key" people and tried to find a correlation. The result was interesting: nothing for the programming test part, etc. BUT, there was a strong connection between "interesting" hobbies and key people. I.E., if you did something like gourmet cooking or sky-diving, chances were good the person would end up being cool. So, now when I interview, I always ask what peoples hobbies are. It seems to work pretty well! --stuart

Well Duh...

just ask for their Slashdot ID, and then you can evaluate their competence based on their comments and their karma ;-)

Basic interview questions

I like to ask some basic questions to get a feel for their understanding before going into any depth. Here's my favorite question and answer from an actual interview: q. What's the difference between TCP and UDP? a. "TCP is from Microsoft. I don't know what UDP is."

Re:I've intervieved a few and...

Those are just dumb questions. Your basically picking someonees brain for how much USELESS INFORMATION THEY KNOW. Your looking for someone fresh out of training with 0 field experience because someone with 5+ years of experiences isn't going to bother memorizing BOOK terminology for interview anwers.

You ask a Sr. level admin what are the file types on unix and he will LAUGH AT YOU.

You ask a Sr. level admin how he runs his systems, how he works under pressure, if he/she is available during odd hours and get a personality feel for them. Find out if they're resourcefull, independant and a good co-worker. Don't play 20 questions and post the answers on slashdot :)

1 Good admin can run 4-5 nt servers, all your unix servers and maintain your business applications, databases and still have time to read his (her) email.

my experience

I've interviewed a few candidates over the last year or so. We tend to do team interviews. We ask some funny questions sometimes, almost as a joke, but often the answer speaks of how they will mesh the team. We have even asked "what's your favorite shell" and "what's your favorite editor" and "what's your favorite os" :)

Seriously though, I would ask about experience with multiple unixes (assuming a unix admin position), backup systems, perl scripting, shell scripting, shared file system experience (we use AFS where I work), "special project" experience (like beowulf clusters, firewalling, etc). Often a sysadmin is already a particular personality...my experience has been that they seem to have a great uniformity of character. Trust your instincts...not very many people will be able to give adequate answers to 3 of the questions above without being sysadmin material. Oh, and another good question..."Describe an experience that yoiu had with a difficult user". this will show if they have a "screw the user" attitude or have a realization that the user is the job.

My test of sysadmins...

We start in the cubicle and have them talk about their current and past job. The kind of things they did. The kinds of machines they worked with. The kind of environment it is.

Then, we go into the technical questions. Things they should be able to explain. Like when you use 'uptime' (or 'w'), what do the three numbers after 'load average' represent? Here. Run a command on this box and tell me how many processors it has, what speed, and how much memory. I run 'top' on a busy production box and have them describe what they see. That kind of thing.

Also, I give them a tour of the datacenter and, while we're in there, have them identify some simple things like cards. Or I'll poing to a Sun E4000 and ask them to tell me, in general terms, about the E4000 server and what it is capable of. I might ask what the difference between the E4000 and E4500 is.

Yes, I also see how they respond to questions they don't know the answer to. But a lot of what I look for is personality. How well they're going to get along with the group and others.

The best question I ever got...

There are two unix machines named A and B that are on the same subnet. Describe to me, in as much detail as possible, what happens when I type "telnet B" from a terminal on machine A.

The "in as much detail as possible" is the key phrase here. The interviewer got to see an understanding of (or lack of) PATH, inetd, DNS, subnetting, TCP/IP, ethernet, etc.

That question, and the discussion we had afterward impressed me so much about the technical caliber of the manger, I took the job.

trick question

Throw in atleast ONE trick question:

"Do you have an expeirence with the Thruman Process on Unix or NT?"

"Ummm *cough* yea they used it breifly at the last company I was at"

"On a scale of 1 to 10, 10 being the best, how would you rate your knowledge and expeirence with the Thruman Process on Unix or NT?"

"Very much so, I would have to give myself a 6-7"

"Do you have expeirence with the Uma Modules to the Thurman Process?"

....

SAGE have an excellent booklet on this

Don't sort through 300 random Slashdot trolls. Join the System Administrators Guild [usenix.org] and get their booklet on Hiring System Administrators [usenix.org]. That should answer all your questions in one hit.

Ade_
/

actual interview question

them: so if (this organization) was a circus, what role would do you play?

me (thinking): what the fuck kind of stupid question is that??

me (speaking): *laff* I clean up the elephant shit.

I think they wanted me to say ringmaster or something.