Are Sysadmins Really that Bad?

tgbrittai asks: "According to Paul Boutin they are merely an obstacle to be manipulated or outmaneuvered. According to Steve Wozniak they are pimps. I've known my share of good and bad sysadmins, programmers and every other professional role out there, and I have to wonder: are sysadmins really THAT bad?" Most times sys-admins are overworked and underpaid and have to deal with users who take advantage of their local IT person, tasking them to fix systems that they callously break. Others are truly worth the name "Bastard Operators from Hell". How would you rate your sys-admin and what things did you have to do to make things run smoothly (or not)?

Are you trying to get us in trouble?

Are you trying to get us in trouble?! It's a damn good thing I'm a subscriber, I managed to block slashdot in our squid cache and drop it in a dns blackhole just before this story went live.

Re:Are you trying to get us in trouble?

I think Woz's post tells us what a lot of us already know: Just because you're "technical" doesn't mean you can be a "high-end" administrator or understand the difficulties/nuances of "scaling up".

It reminds me of many, many, many conversations I've had with programmers, qa, etc, over the years where they tell me what they perceive to be the solution to the problem without really understanding either the long term impact or other factors. [I'm sure we've all heard the "disks are cheap" line when someone has filled their home directory with crap.]

Re:Are you joking?!?

Every conversation I have with this guy is a reinterpretation of "Who's on first?"

Not surprising. If you had told me "I just entered a Pre-approved ECC to move the Learning Center content out" I'd say: "Oh, fine, good for you, what the fuck do I care."

What *is* surprising is that your sysadmin hasn't LARTed you with a ClueX4 over the head numerous times.

Users are by far the worst thing about being a sysadmin. The three B's (Bosses, Bureaucracy, Beancounters) are far behind.

Oh, and it was absolutely right to ask you to keep your voice down. Sysadmins don't worry about bitching users - but they like it quiet.

-Lasse

Re:Are you trying to get us in trouble?

Are you trying to get us in trouble?! It's a damn good thing I'm a subscriber, I managed to block slashdot in our squid cache and drop it in a dns blackhole just before this story went live.

You must be one of the good sysadmins. The bad sysadmins have just been yanking the cables out of the back of the routers.

Re:Are you trying to get us in trouble?

Actually, those would be "brutal" sysadmins.
A truly evil sysadmin would have root access and rootkits installed on all user's home computers already ("here, have this software for home, you'll love it") and have them all zombified under his command. ;)

Re:Are you trying to get us in trouble?

Chaotic Neutral...

Yes

End of Thread.

Re:Yes

I'm sure they must do something important, but as far as helping the employees do work they are completely useless as far as I'm concerned. They come up with policies like "only use IE 5" at a computer security company and withhold privileges from people who know way more then they do, and block all security patches. Oh, and block all the dangerous hacker websites like redhat.com and secureinfo.com.

I once asked the sysadmins why we have 100 people on one printer that always breaks down, and could I be added to one of the many idle printers, and was basically told to go hack the servers and reconfigure the network myself. This was at a huge defense contractor. These were the same guys who backed up gigs of work on the F-22 onto obsolete tapes and then deleted the network drives and then threw away all of the tape readers. Yes, all of them. I was later assigned to reverse engineer what were basically sealed black boxes and re-do the VHDL. That begs the question: "why didn't the buy used readers on ebay?" Good question. Oh, and we had a 5 MB share drive to store all of our work, so obviously we stored almost everything locally. If we reported a problem with our PC they would reformat the machine. Once I told them I had a problem and to not format my machine unless they back up all the data. They said fine. Later I returned to find my machine formatted and my work gone. When I called they told me "we never back up data, it should be on your share drive." They decided something was wrong with my cubemate's computer and snuck in when he was at lunch and reformatted it without ever informing him. He lost years of email and all of his work. After that everyone put signs on their cases that said "do not format this machine." Oh, and our net of 50-100 people was on token ring and none of our apps were installed locally. If one person kicked the cable wrong we were all out of business.

At another defense contractor I reported problems where my machine would lock up for 300 seconds at specific intervals, and was told my problem was impossible and it didn't exist. I reported it many times before finding some lower guy, telling him about it, and he fixed the DNS server 5 minutes later. They never did fix the feature where if I set my clock to the correct time the server would change it to be off by 17 minutes. They also insisted that was impossible and never looked into it. It's not a bug, it's not a feature, it's a hallucination apparently.

Later at that company I requested a laptop with admin privs before leaving on 1-2 week trip, only to told after I got there that they don't give admin privs so that I basically lugged a boat anchor across the country for no reason as I couldn't install the compiler. You should have heard him whine when I said "So in order to use it I guess I just have to reformat it. No problem."

Woz is JOKING, you guys.

From woz.org:

"If my son wants to be a pimp when he grows up, that's fine with me. I hope he's a good one and enjoys it and doesn't get caught. I'll support him in this. But if he wants to be a network administrator, he's out of the house and not part of my family. I tell this joke a lot. Once, a teacher told me that she tells the same one but for a 'teacher'."

Re:Woz is JOKING, you guys.

Yeah, I got my back all up and was ready to write a diatribe (I did RTFA first, however) and found out that this story is a simple case of a lack of reading comprehension. Every time something like this happens, it makes me sad. Especially on a geek news site, where probably a lot of the people have dabbled in programming - they'll be as careful as they need to be to get their code to validate, but when it comes to understanding a natural language, they won't even put in the effort. Then we end up with crap like this. Half the time I'm explaining something to someone, it seems like they just don't get it. (the other half of the time they're raising on-topic objections) :) Maybe I need to dial back my vocabulary for the average person, but I think there's something about logic missing there too.

Boutin has a good idea....

....when he suggests "Treat everything he does as a favor. ". Actually, that's not a bad life strategy - when the waitress refills your coffee right away, treat it as if she didn't really have to - because, really, she didn't! She could have just ignored your empty cup, or waited a few minutes, or whatever.

Same with a sysadmin. When he adds a rewrite rule (done! [thenewsroom.com]) 20 seconds after you ask for it, act appreciative and say thanks, even though that's his job. Because he could have put it off until tomorrow and probably would have reasonable excuses for doing so. (Incidentally, I hosed up this rewrite rule the first time by leaving off the trailing $. Doh!)

Re:Boutin has a good idea....

Thanks for this enlightening post. You could have not done it, and I'd not have replied and wasted more of my day. But it was a favour to the entire Slashdot community. Cheers!

Re:Boutin has a good idea....

No kidding. A little common courtesy and politeness goes a long way. If someone is polite and friendly with me when I help them, I'll put forth an extra effort, and I will remember their attitude when they ask for help again in the future. And it also works both ways. If I take a relaxed, polite, and understanding attitude towards someone who's helping me, I generally get better results. And even beyond that, I just find that being nice is much more pleasant for myself than being angry or impatient.

Chances are that even if you like your job, from time to time you get tired, or stressed out, or just generally annoyed. You don't always know exactly what you're doing, things take longer than you expected, sometimes the tasks just pile up faster than you can take care of them. Why someone would expect that anyone else's job is any easier or more fun is beyond me.

All that being said, some people are just plain dicks, and all the politeness in the world won't change them. I don't know how to make it easier to deal with that, other than to take some solace in the fact that people like that usually are unhappy.

Re:Boutin has a good idea....

The issue as I see it is that a sysadmin enforces rules. SOme people just have not matured enough to handle this. As children they resented their parents for enforcing rules and slowly learned how to become adults. As young adults they resented society/police for enforcing rules and slowly learned how to be good citizens. At school they resented teachers/profs for enforcing rules but learned how to be good students. At corporate jobs they still resent sysadmins/IT and some have not learned how to be a good employee or a good person, thus this tension. Its funny how its only the younger crowd (usually first job) that have this sense of priveldge and are always seemingly pissed at IT because they cant install warcraft (or whatever) on their PCs. Eventually they grow up, or get fired.

Re:Boutin has a good idea....

I think it's *always* a good idea to thank people for their efforts - granted we all get paid to "do a job" but we're not cogs - we're people. Knowing that someone appreciates what you've done is an incentive to do these things because you want to, not because you have to.

Re:Boutin has a good idea....

Yeah my boss keeps calling us resources. I quite frequently remind him I am not a resource like a server, or a software provider, I am a human being.

Now when's Sysadmin Day??

Re:Boutin has a good idea....

Yeah, that's why I always get good service at places I frequent even though I'm a moderate tipper.
Mainly because if the wait-staff looks at my table, I say thanks. I win through being the lowest maintenance patron in the joint.

It's a zen thing. Get great service by not wanting it.

This is the operations crew

We're watching you and we also happen to know that a 3 phase 415V busbar runs dreadfully close to the RJ11 cable which terminates at your office desk telephone. I also recall this cable being reported as damaged (stripped bare by *accidental* friction). However the repair job is in our very long backlog queue at the moment, just after "Halon release trigger malfunctioning in Lockejaw's office".

Please call us if you have any further queries on what we do down here in the basement.

Quality of sys admin is inversely proportional to

Quality of sys admin is inversely proportional to the number of rules they have to work under.

The more red tape an admin has the worse the actual results they will provide

When you take a good sys admin, tell them what you want, give them a sensible budget and ask them to delvier it you will frequently get a great system.

When management try to micro manage, heap them with rules, specify particular components because they read an artical that described it as good or the vendor took them out to lunch you will get problems - lots of them.

Right now I work in a very large bank and some days I think the admins could not find their rear end with both hands and a man page - I've never met them persoanlly so no idea what they are actually like. From otehr friends I have working in banking I know how much red tape they have to work under and I suspect half of the problems the user end sees (bear in mind I'm an ex admin myself and now developer) are caused by the red tape, not by the admin.

Virtual break glass on the root password? 2 weeks aproval before changing anything, even if its trivial? These are the kind of things that can drive an admin insane.

Last company I worked for was a start up - a great place to work for a short period, the admin their was very competant on solaris, windows and linux, had a great system implemented. It didn't start going downhill til a new CEO came in that started to micromanage him (and everyone else). Thats why I got out, same for a few others, the sys admin is leaving when he can find something else he wants to do. Still even with all the hassel he had I still got great results from him, mainly because I respected his limitations, didn't break things, knew what I was doing and helped him out when he needed it. Sales staff on the other hand? With them if it wasn't explicitly on his supported list he;d tell them to take a running jump - because of all the hassel they caused breaking things (the same way repeatedly), ignoring instructions, using unsupported devices or software and then wanting it fixed - and they wondered why he didn't want to help them?

Sys admins are human like the rest of us - overly managed they are stiffled, pissed off they are unhelpful - what else would you expect?

Re:Quality of sys admin is inversely proportional

I strongly agree with everything you're saying. One of the 20 unofficial roles I have at a large bank is Unix System Administrator. I really only spend ~100 hours/year doing system administration, and that's only to deal with something breaking. We have enough work for a full time sysadmin, but we have management who aim to consistently do less than the minimum. I believe the fundamental problem of system administration in any business environment is that you never see the benefit of good results. You only see costs of failures and people running around putting out fires all of the time. A good system administrator tends to work himself out of a justification for a job because there's no compelling business reason to keep employing someone expensive whose benefits to the organization are invisible. Coming in on the weekend to replace hardware, fixing things that break before people notice they are down and recovering files for people who will never admit that they deleted something important are all common sysadmin tasks that are rarely acknowledged.

Micromanagement and imaginary, perceived cost savings create unsustainable environments. Here in a non-technology group of a large bank, we've got a handful of Sun servers attached to an EMC. There are numerous persistent memory errors on the Sun's that could be fixed with a service call and a small scheduled downtime. Well, in a normal environment that is all it takes. However, we don't currently have a maintenance contract. We did have a service contract years ago when the problems started, but maintaining systems is an anti-goal for management - apparently there is no profit in keeping things running. The EMC has been performing well, with the occaisional disk failure that is completely invisible thanks to RAID and automatic call home to get a replacement disk sent out. That's been our key saving grace since we don't backup anything(including production servers).

Unfortunately, this kind of short sighted, unprofessional approach to IT is common in business driven organizations. When everything comes crashing down, as it always will given sufficient time, someone will look at what happened and try to prevent it from happening again. This is the kind of sabatage through mismanagement that leads to the creation of company policies that make it hard for anyone to do their job. Our company has policies that require that system, network, security and database administrators all be separate people. The developers have to be separate as well and can't have access to production systems. There's some very good reasons for all of these policies, but business people can't resist the temptation of hiring one person to do all of these jobs. After all, who better to get things working and fix problems than a developer with root access to everything. It sure cuts down on time wasted in getting authorizations and having meetings.

Been there, done that. Dammit!

I believe the fundamental problem of system administration in any business environment is that you never see the benefit of good results. You only see costs of failures and people running around putting out fires all of the time. A good system administrator tends to work himself out of a justification for a job because there's no compelling business reason to keep employing someone expensive whose benefits to the organization are invisible.

You said it. I was one of two Unix SAs supporting a few dozen servers for which several hundred users depended for their jobs. If something went wrong, they called and, just like magic, things were fixed. They loved us and they loved the application. The worst thing that could happen would be a server death and when that happened, we'd call up the manageer of the affected group, ask them to have their people save their work locally and sit tight. Out of the closet would come a pre-configured replacement server. We'd plug it in, restore data from one of our three redundant back-up systems, and have those users up and running again in two hours, max.

I loved the work. Absolutely loved it. Because this was a government job with generous paid leave when one of us would be gone, having two of us meant there was always coverage and no downtime. Given that our users brought in 10s of millions of dollars a month, we were a paltry and perfectly justifiable expense.

Our problem was that nothing ever went wrong. Our big 'ol rack of servers hummed along with no drama and whenever the boss dropped by, he'd likely see us plodding through something routine like adding a user or checking system capacity reports. Every few days, we'd get bored and actually walk around the cube farm of the users, stick in our heads, and ask if everything was ok, can we do anything to make things work better? Our users loved us; our bosses didn't even seem to know what to write on our evaluations.

The Windows servers on the other side of the datacenter? Holy Cow, did those guys have the drama! Things were crashing all the time (We're back in the early NT days, mind you.) Whole populations of users suffered critical amounts of downtime. The admins put everything back together, of course, and were lauded as heroes because they had fixed the big, bad problems that had killed so many people's productivity for so long. They were HIGHLY visible to management. They got awards for fixing things. They were heroes.

Us Unix admins were those two people who sat over in the corner and never seemed to actually, visibly do anything.

You can see where this is leading, right? The Windows server side and the Windows front-line support side needed warm bodies, so I got thrown off Unix and into a GUI world I neither wanted nor understood. (Don't get me wrong, I've done the Windows work for years and I love helping people, but I'm not in love with the OS I now use and support.) Later, the other SA was tossed and our servers virtualized on mainframes. The number of SAs was cut to the bone and beyond. Virtualization was a nice concept and it works fine, but getting something fixed when it breaks is now a major red tape experience for our poor (former) users.

Fires to put out mean that firemen get chances to become heroes. Safety engineers who inspect your business and show you how clean the grease traps so nothing actually catches on fire are just needless expenses to be cut as soon as possible.

The moral is: Be a fireman. I figure they get more women, anyway.

Re:Been there, done that. Dammit!

The real moral is that if you want to be a valuable geek, you have to learn enough people skills to make sure other people know. I've got a couple of decades of professional experience under my belt and am an expert in several areas, but the most valuable experience I've had professionally comes from working in a large company with a good number of untrained monkeys.

There are a lot of people who can't tell the difference between a seasoned professional and someone who would have bought a computers for dummies book if they were literate. Some of these people will be promoted into management to keep them out of the way of people doing the work. Being able to interact with people on their level is an incredibly valuable skill. It's nice to work with intelligent people who know what you do, but not everyone gets that kind of dream job. Basic communication skills are important, even if you feel like a retard when you're doing what is expected. If you don't feel like a retard, you're probably not going to effectively communicate with the business people. =)

For example, I'll send out emails to users, managers and the VP to let them know that a disk on the EMC failed, switchover to one of the hot spares occurred without incident, the failed disk was replaced and transitioned back into the array without issues and with no more than negligible performance degradation to the systems and users. No data was lost and we're back up and running. This happens once or twice a year.

If you know anything about EMC arrays, storage systems in general, or how to get your VCR to stop blinking 12:00, you probably realize that I didn't really have to do anything other than be aware that something happened and let the field service technician do his job. I've spent my whole career learning about technology so I am perfectly capable of doing all of the maintenance myself, but in this kind of case, I just need to let someone else do their job. This is not exactly rocket science here. However, people who don't get the technology see something like this and think "Huh, I guess something broke and now it's fixed and everything's good. Good thing he knows what to do because I wouldn't even know who to call or what to say to them." Most of the people whose opinions matter have no idea what you do.

There are a lot of arsonist-firefighter types in IT. You can be just as valuable as them without losing any shred of decency as a human being. Just let people you help know to let your boss, your boss's boss, their boss, and anyone else they know how incredibly helpful you were. Chances are that they asked for your help because they needed you to do 10 minutes of work so they could avoid trying to spend weeks trying to figure it out themselves and making it much worse before it got to you. Most people will be willing to spend 60 seconds to send a quick email to help you out.

Re:Quality of sys admin is inversely proportional

The first IT job I took after getting out of the army was answering phones on a help desk for a retail company. We had to support about 1,500 users.

Not long before I started that job, the company had hired a new "Director of End User Technology" and this guy was sharp. His primary goal at the time was to straighten out the cobbled together mess of a network that had haphazardly grown department by department. The place was a real mess and the network ran like mud.

Over a period of about four years, we standardized our PCs and laptops, physically consolidated the servers that were spread all over the HQ building, corrected the messed up cabling, centralized administration, built a training room and implemented a number of classes, etc. It was a truly exciting and fun place to work and virtually everyone who I started out with on the help desk eventually learned, got certifications and moved into administration and/or engineering. When I had started there we had a real mom & pop shop type feel and very little oversight. All we had to go on were some clearly defined goals and a directive to "get things fixed."

We consistently accomplished our goals. Within the first couple years we had fixed the network and made it into something useful. The consequence was more use by upper management and as you might expect, more management from upper management. Every time we met another goal, the more visibility we received. The more visibility we received, the more layers of management they installed above us. Every layer of management installed made it harder and harder to actually get anything done, basically because each new layer of management knew less about IT but more about "managing.".

I guess mostly I'm just whining here, but eventually most of us who had built the network quit. They 'managed' us right out the door.

Re:Quality of sys admin is inversely proportional

Quality of sys admin is inversely proportional to the number of rules they have to work under. The more red tape an admin has the worse the actual results they will provide.

Very true.

I'm a sysadmin in a DoD Classified network on a USAF Base ( LOTS of red tape) and the first rule is security (which it should be). That pretty much means lock everything down. Some examples include: lock the USB ports, prevent writes to CD-RW Drives, prevent writes to DVD-RW Drives, audit everyhthing (PL2), prevent printer installs, software installs/removal, lock down screen savers (executable code), password changes every 62 days, approved software installs only which usually means we are lagged on releasees, etc, etc, etc. Some of these are silly, yes, but I don't make the rules. The "red tape" is mandated to us by the Air Force.

All this red tape creates a very unfriendly user environment where the users frequently are annoyed with the admins because they can't do something as simple as copy data from a classified PC to a classified laptop for a presentation. They have to track down an admin to do the copy for them. Paperwork must be filled out and whitnesses present. They may not have access to files due to security permissions. Won't delve into the requirements here but it has to do with employees from different companies all working the same program who potentially have access to each company's proprietary information. I can go on and on, but the bottom line is red tape creates a very unfriendly user environment where the users frequently claim the sysadmins "don't know what they are doing", which isn't the case at all. The users are deliberatly not allowed to do what they are trying to do. However a majority of the user community thinks us admins make the system painful to use on purpose. Not the case. and they frequently take out their frustrations on us.

Everybody is overworked and underpaid

Don't cry me a river about that...with the exception of upper echelons of management, I'd say most people do more for the company than they get back as a reward for their work.

I've been on both sides of the fence, I've seen users that put every piece of software they can find on their machine, then come calling when they break. I've been blamed for doing something to break a printer, about two weeks after I was there to swap a monitor.

On the flip side, I've worked in places with a tiny server share to store important data and an IT staff that doesn't really guarantee it'll be backed up. So we ended up having to work around the IT staff in a lot of things. It was easier to cobble together something that we can guarantee is backed up AND that has enough space for us than to go through the reams of paperwork to get more space and justify some sort of improved SLA.

In fairness to the IT folks though, a lot of the people working IT are just trying to feel their way through the system that was put in place before they started, and they think it's just as stupid as the end users. But they lack the power to change it, and their bosses don't want to.

Re:Everybody is overworked and underpaid

But they lack the power to change it, and their bosses don't want to.

That brings to mind my first rule of systems administration: Give me the authority and the resources to prevent the problem and if it breaks anyway I'll work 20 hour days to fix it. Get in my way and stop me from preventing the problem and I'm headed home at 5:00 whether you're in a frothing panic or not.

Most places I've worked liked the display of initiative and steped back to let me do my thing. They liked the results too: 20 hour days were very very rare.

Not really...

Not really..... now what was your username again?

*clickety*

http://www.theregister.co.uk/odds/bofh/ [theregister.co.uk]

Lack of experience

I think the new "fleet" (if I could call them that) of sysadmins are too inexperienced and are often thrown into a wild west of "our infrastructure works like this!... With an infrastructure that many times hasn't been planned out too well, is highly misconfigured, is a nightmare in progress. Often those sysadmins will have to adjust to someone else's tailored system and will fail miserably... I've seen it for years on end, horribly designed systems with no documentation, horribly managed systems butchered to perform a task. No two systems will be alike and I believe its this same scenario which makes or breaks an admin... However with the newer sysadmins coming around, and I've seen plenty in the past 3-4 years, they're inexperienced... Running Linux @ home or your own personal webserver does not make you a bonafide sysadmin. At least not in my little space... I know admins who strictly know perl... Good for you. Now go fix this legacy system which by the way doesn't have perl on it, and you're not allowed to install perl... Would you know how to do so in say awk and sed? To me a sysadmin knows things from the core up, not from a yum install *something*, apt-get *make-me-look-nice*, or whatever other command. Just my two centavos

Re:Lack of experience

I dont disagree with your statement here, necessarily, but it does sound to me like an issue of pointing out a problem, without really offering any notions for improvement. You say that new sysadmins are too "inexperienced" and dont necessarily know enough about Legacy systems. Well... how would they? I mean, if you've worked on any mainframe systems, you'll know that knowing one set of commands doesnt do you ANY good on the very next mainframe you might be forced to work on. And how exactly does one become an "experienced" sysadmin? Go to sysadmin school? Really? Sign me up!

It seems to me that things are the way they are because... well, they have to be. When an old sysadmin leaves, you're not going to be able to replace him with someone who knows everything about your current infrastructure, and happens to have niche knowledge of all of your various legacy machines. If such a person exist, chances are very high that they're currently still employed somewhere else, or are about to retire. Employees dont stay in circulation forever. Eventually mass amounts of experience starts falling out of the market, and has to be replaced with "noobs".

Im not saying huge companies should necessarily be hiring inexperienced sysadmins. But someone has to, or inexperienced sysadmins can NEVER become experienced sysadmins. Im fortunate, in that I was hired on as a sysadmin at a University, during a complete infrastructure rebuild. So while Ive been forced to learn a passing familiarity with the mainframe systems, it's mostly been to help usher them out entirely. And Ive been, for the most part, at liberty to build the new infrastructure around her to my own personal standards and benefits, meaning Ive got a pretty good grip on things. Gradually, I run into problems that I cant solve with a simple script, so Ive been forced to learn things like sed and awk, as you mentioned, more and more over time. And even those, btw, arent a universal solution, especially if old IBM-era mainframes are involved.

Even if what you're saying is the problem, if sysadmins with "not enough" experience for a particular job are being thrown into them... there's no real solution for that. I mean, if you draw up a requirement for all of the systems you want a sysadmin to know, chances are NOT good that you're going to find someone who A. Meets all the requirements, including experience with all of your legacy systems at your company B. Lives nearby or is willing to relocate to where you are and C. Is looking for a salary exactly where you're offering it.

Re:Lack of experience

The solution is so painfully obvious that no one will ever implement it.

Business people need to look at more than one little line item on a budget. There are a lot of jobs that pay $50-80k for a sysadmin. The vast majority of day to day things can be done by one of these people. When they get stumped on legacy stuff or something really weird, they end up spending a lot of time spinning their wheels and have a hard time getting the problem solved.

The other option is to hire the $150k sysadmin who has tons of experience and makes the hard problems look easy. These are the kinds of people who you can give 3 months to solve a problem, or you can hire a team of 5 people to work for 20 years on the same problem. If you put it in that perspective, the money is well spent.

Smart business people look at numbers and know that $150k is more than $50k, and also know that if they yell loud enough about the $100k they saved, some of it will end up in their bonus.

The thing that seems obvious to me is that you hire a bunch of the cheaper people who can do all of the normal day to day stuff, and you also hire a guru who gets all of the impossible tasks. The less experienced guys learn from the guru and the guru doesn't spend 99% of his time doing tasks that would be better suited to a college student or a shell script.

Of course, companies don't like this idea because HR people don't want to believe that one person can be worth several times as much as another person who is referred to with the same type of job title. In HR there are no gurus, so the concept is completely foreign. After all, if someone was inclined to be a guru in any field, how would they end up in HR? =)

Re:Lack of experience

The thing that seems obvious to me is that you hire a bunch of the cheaper people who can do all of the normal day to day stuff, and you also hire a guru who gets all of the impossible tasks

And that seems to me to be the key thing: inexperienced people need to work along side experienced people. That's how people get experience.

Why is this such a hard concept for people to understand these days? It's like there are two camps: either you think companies should hire all-knowing experienced geniuses or you think companies are better off hiring a small army of inexperienced guys.

Throughout history, in pretty much every trade, there's been this idea of apprentices. There's been the idea of "working your way up the ladder". The idea is pretty simple: you put the new guy in with the experienced guy, so that the experienced guy can pass on his knowledge and the new guy can get up-to-speed. Over time, the new guy learns enough to take over the small/easy portions of the experienced guy's work. The experienced guy gets to avoid the crap-work, and the new guy gets experience. Over more time, the new guy starts becoming an experienced guy, can take on more complicated problems, the experienced guy can keep focusing on higher and higher-level problems, and it keeps building on that model until either the old experienced-guy or the new experienced-guy move on to something better.

Re:They often can be

This comes from hiring the cheapest person HR can find.

My SysAdmin is my Boss

And he is the best Boss ever. He even reads /., which should demonstrate how cool he is. He certainly wouldn't do anything bad, like access my computer, log on to Slashdot, post an article telling the world how awesome he is, and then give me a warning to secure my computer (and change my /. password).

Sys Admin frustration

I currently work as a Network Admin for a large retail company. I started with this company as a store clerk, then moved to the helpdesk, while i was in college. This helped me learn patience and how to be polite with everyone no matter how annoying, wrong, irate the "customer"/"employee" is. I look at some coworkers who have no clue how to handle talking to a customer or a user needing help and give them lip every chance they get. I understand the frustrations of having petty work assigned to you by a VP level person that interrupts your day and workflow. All the time i have important time constrained projects interrupted by those "important" people who have to have some blackberry/treo/etc problem fixed asap. I have to drop whatever important task I am on and concentrate soley on the happiness of this one person. Ultimately thats what it comes down to i have found. Although i get my work done and i am thorough on all mky projects, I am not known for that, I am known for always being the nice guy who helps out the Execs and their exec assistants, and honestly that puts me in better light than anything else. Sometimes the interruptions are extremely frustrating, but when the execs are happy, everyone is happy.

I hate dealing with Sys Admin

Its not that there bad people. Most of them are pretty nice if you talk to them over lunch. The problem is they are so constrained by what they can do that they are very frustrating to work with. I must say they also hate working with me for the most part.

The problem is that I'm somewhat tech savy. The sys admin don't like anybody trying new things. Their management likes it even less. Do one little thing or install one little app and if you have problems your on your own. Doesn't matter if your laptop explodes, they'll blame it on VMWARE or whatever you happen to be running.

It wasn't always like this. In days of old the Sys Admin were local and reported into the same groups they supported. As such they knew what we were working on and would help out. Management would support this because it often lead to increased productivity or reliability. But at some point a bean counter decided we needed a corporate IT organization.

Once you decouple the support from the groups they support you end up with apathy and endless rules. Also to get the groups to try anything new you have to weave your way through a bureaucracy. You also end up with smaller and smaller IT groups because their contributions to the end product become harder and harder to trace. If a business unit needs to cut costs the first thing they look at is horizontal organizations outside their own structure. Its a lot easier to cut an outside IT guy than a developer working on a product.

Things look to be taking a turn for the worse. Some of our IT is now going to be out sourced. To me this is equivalent to saying I now fully support myself. I can just imagine trying to convince some contracted person in India that I really do need to have VMPlayer installed on my Windows laptop....

Re:I hate dealing with Sys Admin

This is spot on. I am that sysadmin you talk about. i work for a large software company - however, I was hired specifically to support one smaller office, as the IT group couldn't (or wouldn't) provide adequate support. Things were great for a while - I learned the quirks of this particular office, setup new systems and generally made the work environment better for those that work out of this office.

Thats when a new manager, and IT overlords stepped in. Now I have to do everything 'by the book', even when 'the book' doesn't mesh with what we are doing here at this satellites office. My life is now a hell of process, procedure, and meetings - and very little actual work is getting done.

What does this lead to? Developers going 'out of band' to get stuff done - purchasing hardware on credit cards, not using authorized apps, copying large files around the WAN when stuff should be local, etc. All because they can't get a slice of my time to help them with a correct solution.

Everyone here is frustrated - myself the most. I *want* to provide the best support I can, but I'm now hamstrung by process and management, whereas before (when the developers/local managers were happy) I wasn't.

I think most sysadmin jobs are going this route now, excepting the startups (and they will, as they grow). Sysadmins are a commodity now, they aren't viewed as adding value.

Re:I hate dealing with Sys Admin

The problem is that I'm somewhat tech savy.

Yep, taht is the problem. That is right, you are the problem.

Do one little thing or install one little app and if you have problems your on your own.

This is generally because most places have rules against users installing apps on their own.

The problem is they are so constrained by what they can do that they are very frustrating to work with.

They are constrained by management and good administration. If you are frustrated that you can't do something, either you need to take it up with the people who set up the rules or you need to rethink what you are doing, because it is going against policy.

I can just imagine trying to convince some contracted person in India that I really do need to have VMPlayer installed on my Windows laptop.

It should not be hard if you do need it. You should be able to say "I can't do my job without it" and that should be that. If you can't do that, then you probably don't need it.

Re:I hate dealing with Sys Admin

You've hit on a few of the key points in your post. However, let me address your complaint about sysadmins not liking people trying new things. In general, we admins LOVE new things! Take a look at all the gadgets around me or software installed on my workstations, and you'll see the truth in that.

However, there's also a supportability issue. If I have five users I'm responsible for, then I'll happily accept five different machines. If I have 30 users, then I don't want 30 different builds and application bundles. If I have 500 users (or even 100), then I cannot AFFORD to have variance between machines, if I'm expected to support them.

You want a program installed? If I'm going to install it, then I will have to make sure it won't interfere with the existing software, and then I have to keep track of the fact that your machine is different than anyone else's. If someone else wants a different program installed, same problem, squared. Alternatively, I can give you admin access to your workstation or laptop, but then I can't guarantee anything about that machine anymore, and can't support it.

The third alternative is to put in a formal request to have the software added to the official bundle, or at least put on an 'allowed/approved' list. That's the best solution, but also the most onerous, bureaucracy-laden, time-intensive one, as you well know.

Mostly, it's a matter of (a) scale, (b) supportability, and (c) accountability. If your system is strange and nonstandard then when it breaks it's easier to say, "it's " than explain the reasoning behind, "because you have installed, I can't help you."

I feel your pain, but there is some valid reason behind it.

Is X really that bad?

Replace "X" with any profession, and the answer is the same: some are and some aren't. The professions with high barriers to entry (i.e. medicine) tend to root out some if not most of the incompetents or otherwise poorly qualified, but some will still slip through. The same is true of sysadmins. They obviously exist for reason -- maybe the article writer should ask, "What would a world without sysadmins look like?" For large organization, the answer is "chaos," and they would quickly re-implement the same positions now being mocked.

Sysadmins are the greatest people ever...

Pay no attention to the systems administrator part of my job title...it's just a standard honorific. >_>

Before I launch into this, it really seems like they define good and bad by their customer service skills, so that's what I'm addressing by "good" and "bad", not so much their technical knowledge.

In my experience, the problems with sysadmins tends to be that with the ones that lack the ability to understand the user. This is what people refer to as the "IT mindset" where the user is the enemy and is doing whatever they can to make IT's life more difficult. In some cases, this is very true. There ARE abusive users out there. However, most people simply want to do their job, and their job is NOT getting these machines to work right. Getting back to the "understanding the user" thing, I find a great many sysadmins have no empathy for how a user feels when their machine has gone down, and why would they? When has a sysadmin ever really felt the panic and/or frustration of having a machine crash and not having the first clue of how to fix it? We KNOW what we should do, and while we'll be annoyed at the extra work, we're (hopefully) never flailing around blindly...or if we are we're careful never to show signs of it. A user's machine goes down and they have no idea what to do. They panic, they worry, they don't think logically...they immediately run to the nearest person who they think can help them and oftentimes get the look of "Why should I?" or "Can't you see I'm busy right now?"

Again, that doesn't mean there aren't people who don't actively try to bypass what they SHOULD be doing to get the problem they caused looked at immediately because they think they're more important. However, I think the sysadmins that most people complain about are the ones who let the handful of lazy/abusive users jade their dealings with the ones who simply want to do their job and go home.

However, I find that the "bad" sysadmins are about as common as the truly abusive users. They stand out in your memory so it seems like there's a lot of them, but they're actually far from the rule. YMMV, of course. After all, in the course of a day three or four people might stop to hold open a door for you, but the one you remember at the end of the day is the idiot that cut you off on the highway. Human memory is a funny thing...

Admins are not the pimps, they're the heels

Admins are literally wedged between workers and management.

You, as an admin, get orders from management how they envision the network security to be. You know it doesn't work that way and will only create an obstacle for the people you're to protect, but you will do it anyway. Because the guy you knew from the day shift one day took one such memo and trotted upstairs to the brass.

He hasn't been seen since.

So you do what you're ordered, block non-corporate mail accounts, block porn sites, block ebay, block... everything. This is usually when one of the middle managers complains that he can't go online anymore, which turns out as him being unable to access ebay anymore which he needs for ... umm ... his quarter years report (yahu, sure), and if it isn't reactivated IMMEDIATELY, you're in deep dung.

It escalates up to the top brass, you get said pile of manure onto your head for not cooperating with middle management and you now have to work out a plan how to block ebay without blocking it. Sounds impossible? I know that. You go upstairs and tell the brass. Can I have your stuff?

Then you head down to the cafeteria for some coffee. Coffee good. Coffee lifeblood. My precious. But you forgot your fake moustache and the noseglasses, so people immediately recognize you and start asking what's wrong and why they can't access gmail and gmx anymore. You explain the brass note. Which causes them to tell you in no uncertain terms what a weenie you are, because they need mails from a contractor that the corporate top security firewall won't let pass because they are deemed insecure attachments and how the hell they're now supposed to work.

Need I go on?

Are SysAdmins that bad? Depends on who you ask.

Several others have already said that SysAdmins are only as good as the rules and management that constrains them. Then again, there is the personality issue.

I am a SysAdmin myself like many on Slashdot. However, I do SysAdmin work on two different levels. At my day job, I manage gigantic enterprise class data systems with clustered servers for everything from distributed processing to my Oracle 10g RAC cluster. However, I also do work on the side in my spare time for small businesses and friends in the area. I do everything from some simple web development to distributed networks for file and application sharing. I've been given compliments and complaints but the compliments far outweigh the complains.

What I have heard most and that I like to hear is that people like to deal with me. They like to have me answer thier help desk calls because they know it will get fixed correctly and as fast as humanly possible. I like having that reputation and professional respect. Because of that, I don't have to fight with a user or management when I say I need time to figure out an issue or stand up a system. Does that make me a good SysAdmin? I dunno. I think it makes me a good employee. Then again, I get the same compliments from my small business customers and friends who would rather call me for help with their DSL account or a piece of troublesome software than any help line.

Given that, I think that a SysAdmin is an employee just like everyone else. Because of that, we shouldn't be venerated above others even though we are an employee with a special job. A SysAdmin allows other employees to be productive. If the SysAdmin isn't doing the job they have to do, then the company as a whole suffers. I suppose this is where the 'root is god' can get out of hand. When an entire company's infrastructure depends on the work of a few people, that's a high stress deal. Sometimes it gets to people. Bottom line though, we are all employees and just like the loud guy at the water cooler that nobody wants to hang around with, if we aren't profession and approachable like other employees, we are hurting ourselves. SysAdmins have to be computer geniuses, we have to be business oriented, we have to be people people and we have to be avaialable and approachable. It is not an easy task, believe me, I know! However, we all need to have a certain degree of professionalism when dealing with our customer base (users). We SysAdmins are our own downfall. The poor perception by the slobbering masses of users is our own fault. We can change it. While we do understand that our companies would not survive without us, it is not our place to make it so painfully obvious. The users don't care how great we think we are or even how great we are. They just want thier problems fixed quickly so they can get back to being how great they are. If we can just appease that desire from the users, I think that's what would make a good SysAdmin.

Boutin's article.

Boutin's article has five points:

Say hello. Even when you don't need something.
Don't question what he does all day
Fill out the stupid request form
Treat everything he does as a favor
Never forget he can read your email


What this boils down to is:

Treat him like a human being,
follow policy and procedure,
appreciate his work,
and don't talk about about him.

I guess people forget that SAs are people and employees too and that they work under constraints placed on them by upper management.

IT is a service industry

As a person who worked my way through school waiting tables and bartending, then dropping out in the *typical* SysAdmin fashion, I tend to liken my department's level of service to the level of service one would receive at a restaurant.

Heavily corporate restaurants make their customers sit through a whole song and dance about the restaurants offerings and their associated flair. Heavily corporate IT makes their customers (fellow employees or clients) wade through a song and dance about red tape and process.

Mom and Pop restaurants allow more freedom in day to day management of the customer experience, likewise startups do the same.

Greasy spoons with the head waitress who can run the floor and cook the food and do the dishes while balancing the books do it the head waitresses way...

You can draw the parallels anyway you like.

The real point is, as a SysAdmin, I try to keep in mind that me and my department are providing a service to our clients in whatever way, shape or form you want to define them. Without clients, while there may be considerably more time for Nethack and Slashdot posting, there would be no job.

As a DBA and sys admin, I'd have to say...

We are not "bad" at all. We are doing our jobs. What are our jobs? Our role is to keep our systems operational and secure.

Think of it in terms of roles.

Sys Admin: Protects and defends the infrastructure of your company. Prevents people from shooting themselves in the foot. Enforces good security policies. Identifies poorly performing software and forces its developers to improve it (or get shut down). Keeps your systems patched and ready. An iron fist in a velvet glove. An enigma wrapped in a mystery. A big, sexy man!

Programmer Type 1: Cooperative with sysadmin. Tries to write solid code. Doesn't break stuff. Often has a good rapport with sysadmin and finds, mysteriously, that his jobs get run on time, every time. Filled with the Tao.

Programmer Type 2: Bastard child of Peter Lorre and Marty Feldman (with the voice and the eyes). Doesn't care about correct practice, only what he can bang out in an hour. Takes ridiculous shortcuts, risks crashing servers and services. Source of all memory leaks. Tries to be clever and fails. Mortal enemy of all sysadmins and Type 1 programmers!

User: Whirling dervish of chaos in an otherwise orderly world. Between downloading P2P apps, questionable freeware, and trojan and adware corrupted hacks of popular programs, spends time inviting the wrath of the RIAA and MPAA by sharing his entire music collection from the main file server. Browses pr0n instead of working. Plays solitaire instead of working. Cries like a little girl every time he's forced to comply with official policy. Complains bitterly about those nasty sysadmins. Secretly wishes he was a pr0n star and has been stalking Shelly down in accounting. She'll mace him in the cafeteria later on in the week.

I vary depending on the user(s)

There are some users that I'm sure would comment that I am extremely friendly and helpful. Actually, probably quite a lot of my users, as I do enjoy interacting with them and discussion various issues (partly to be social, partly to avoid recurrence).

The people that might find me antisocial are:

• The ones that assume lunchtime is Q&A period: I generally avoid on-site lunches for this reason. I like to eat too, and I like to relax/read during my break. If it's 12:05pm, and I've got a book in my hand and a sandwich partly in my mouth... I will not be friendly when it comes to answering questions, especially if they're annoying non-work-related (home) ones.
• With the above. When you've watched me work through lunch and it's not 1:30/2:30 and I've not yet eaten or had a break... it should not be particularly surprising that I don't want to check out your "little issue" when I'm done
• Do-it yourselfers: Staff that display a certain amount of technical knowledge combined with restraint are great! If a user can fix minor problems for himself and others then that makes my job easier. If the same user accidentally makes a mistake, no biggie. However, the users that are gutting their computers, installing unauthorized/illegal software, and other such things... grrrr
• Impatience: Everyone wants their stuff fixed first. Quite often one user will be (literally) breathing down my neck while I'm up to my elbows in another issue/machine. Patience and personal space are important things.
• Circumvention: We have a "ticket" system. We have a huge amount of users/systems. For non-emergencies (server down Vs "My printer defaults are wrong") it does not help to circumvent the situation by calling me directly. It is less helpful to call my supervisor. This causes other people to be PO'ed at me for being diverted from their (usually equally if not more important), legitimate tickets. It makes me PO'ed at you. It also means I have no ticket to close indicating all is done, and no tickets letting management track that I've actually been working.
• Following instructions, and being a little independent: There are the occasional users that need to be told constantly how to deal with a particular scenario. They aren't stupid, as they demonstrate an intricate knowledge of their own domain, but somehow manage to forget simply instructions on how to fix annoying issues. For example, if the server has been down (lengthy power outage, or more recent at one site, drive replacement), reboot your machine before calling for help and requiring a 30 minute drive. (some site are up to a 2h15m drive).

See, the above make me seem pretty grumpy, right? But the truth is that most days are fairly pleasant for both myself and (judging from feedback) for my "clients." However, there are always a few people that can magically manage to rain on a sunny day. Secretaries are often both the best and the worst. Some are obviously in their job because of wonderful PR skills, and manage to be extremely friendly, and, more refreshingly, honest (they can admit when they have messed something up, or don't know how something works). They also often have candies on their desks :-)

But trust me, anyone can have a bad mood after being 2-3h late for lunch and when running a full day without breaks.

Food

From TFA:
How do I get my sysadmin to do anything?

Very simple.
Cookies, brownies, pizza, etc. I've worked as a sysadmin, so I know all sysadmins like the ones with the little dark chocolate chunks in them.

Well 3 years in support...

After 3 years in support for a fax solutions provider, I encountered what I would estimate to be a 5% level of competence. Meaning 95% of SA's I talked to don't know their ass from their elbow. I talked to this one admin of a windows network that didn't know what Active Directory is. Let me tell some more anecdotes.

Now, you might say that they're probably underpaid and overtasked... the underpaid and overtasked guys were actually usually the competent ones. I've had Windows admins call me, support for their FAX solution, for EXCHANGE support. Now I mostly handled Linux calls, My MSEXCH knowledge is very cursory... this guy got pissed and I basically referred him to a MS KB article on what he needed to do. Incompetence is so prevalent in the SA job role it's really just insane. It's gotta start somewhere above them though, cause management made the incompetent decision to hire this incompetent admin. So it's sort of a chicken/egg.

I also loved the admins who were all like "we HAVE to get a linux solution! linux is so rock solid! I don't really know how to use it or even log in to it, but it'll be the proverbial rock in the data center!" Then when it breaks here I am phonetically spelling out commands like cd and ls, while they enter the wrong kind of slashes (if they figure out how to log in first). Then when they enter the proper command finally they're like "I don't think it worked, nothing happened."

Another herd of sa's that were awesome were the ones who either had a language barrier or thought it might be impolite to say "no" when you ask "Is the XXX service running?" It makes for a really fun game where you just have to guess whether they really mean yes or no.

Well I won't keep boring you with my mundane tales of the greatest SA's of all time cause I think you get the picture.

They Can Be

Once upon a time, in a land far-far away, I ran the IT department of a medium sized company. We weren't so big that I was the CIO, but I did repot directly to our corporate president (who reported to noone else, since he owned the company). A few hundred seats total, offices in 14 states, most of which were home offices, that sort of thing. I had a staff of eight, including four first-line phone technicians, and on the whole the entire operation ran quite smoothly.

The only times I had major problems, or heard of major complains, about a system administrator was when they started making one major flaw in their perception: that IT was there to do things in a way that was best for IT.

When a system administrator begins to believe that their entire function is self-serving, that they are there to support their own operations, that's when I've seen things go bad. Regardless if you are an IT consultancy firm, or an internal IT department, the sole purpose of IT is to play a supportive role in the organization. It's important to recognize that if IT didn't show up for a week, things would probably be okay (backup tapes would *really* need to be changed), but if the sales and customer service departments didn't show up for a week, we'd be damn near out of business.

I always tell people to think about their product. What is it you produce? System administrators, software engineers, they produce things that let *other* people get their work done *more* efficiently. If it's difficult for the sysadmin or developer to do, who cares? They aren't there to make life easier for themselves, or to devise some system that's perfect on the whiteboard but impractical in the real world - they are there to bring practical, cost-effective efficiencies to their end users.

Now, if you have a guy going around unplugging peoples network cables at the switch because they pissed him, fire that guy and hire a professional.

My $0.02.

I love our sys-admins

I love our sys-admins. They are the most competent, noble, intelligent, and good looking folks I have known or ever will know.

May I have more space in my home directory now, please?

Bad admins versus good admins

There are a lot of bad sysadmins out there and there's no good way to tell them apart, from a manangment prospective.

A bad sysadmin looks very busy all of the time so management and co-workers think that they are busy and important.
Things are always breaking and they come to the rescue. Things are down for days and through their heroic efforts, (cough, reinstall Windows, cough) things are back working.

While good sysadmins are proactive and very little breaks or goes wrong. They remain calm during user's crisises (because panic never fixed anything). They are seldomly seen by co-workers and management. They do things like scheduling down time when the system is working fine or nagging users to do thinks a "better" way when the old way worked fine for the users.
  No one see them fixing much, and nothing ever breaks, and the network is never down, so they must no be very important or valuable to the company.