How Do You Document Technical Procedures? 401
ChadDa3mon writes "I work for a large MSSP type operation and we deal with a plethora of vendors, versions, and .... skill sets. We're facing a critical problem as we grow when trying to deal with these varying degrees of technical competency. The end result is we're getting to the point where we have to document every procedure and process, no matter how mundane or 'common sense' it may seem." How, ChadDa3mon wants to know, can complex skills be documented to account for various users? Read on for more details of what he's seeking.
"I've got a picture of how I'd like this to work in my head, but I can't find any software out there that seems to go along with it. I'm a big fan of keeping things simple, so I'd like to start with high level overviews. Each step in the process would be a general statement like 'Look for valid traffic on the monitoring interface'. For those who already know what 'valid traffic' means, it's easy to follow. However, if there was someone who was unsure what it meant, there would be a link they could click on that would pop open a new window (or something similar) explaining in detail what we're looking for and how to find it. It's my hope that using this process, people aren't just blindly running commands, but gaining an understanding into what that command is, and why we use it, what to be aware of, etc.This seems like a job for a flow chart, but I don't like the setup of any of the ones I've used, such as Visio. It could also maybe fulfilled by a wiki, but there's so many out there I don't know where to start. I have to assume I'm not the only person who's facing a problem like this, so I'm hoping someone else out there can make some recommendations."
the easier to edit the better (Score:3, Interesting)
If there's a mistake or something in your docs are unclear, you want the guy using the docs to be able to fix it right on the spot. For that reason use a wiki, no-one is going to fire up Visio or whatever and diddle a flow char tin the middle of a call.
I assume the people using it are phone monkeys, make sure to track who makes improvements so that it doesn't penalize the guy who takes a minute to fix something but then their talk time suffers and they get bitched at.
"and add it in the computer file later on" (Score:3, Interesting)
That's always the catch, isn't it? I'm sure we've all seen the printed instructions with more handwritten addendums and notes in the margins then there is printed text.
Re:bad (Score:5, Interesting)
Exactly. Also, do not forget that the best way to ensure you will never be promoted, is to make yourself indispensable at your current position. As all career counsellors will say: on any job, you should be training your replacement from day one.
Graphviz (Score:2, Interesting)
For simple procedures, I just use Powerpoint, and have extra, separate graphs when a particular task can be expanded to a subgraph.
If anybody has any other graph packages to recommend, I would really appreciate upgrading.
Sharepoint works.... (Score:3, Interesting)
Re:Make it simple, or you won't do it... (Score:3, Interesting)
Put enough detail in the document so that you can hand it to a marketing droid and have them successfully complete the procedure.
I'm not sure if this is what you mean, but it seems to be similar to a process I've been using for documenting processes for less technical users (usually QA testers, operations people, or business analysts). What I do is write up a quick and dirty document that I feel covers the basics. Then I get together as many people who will need to use the procedure as possible, give them the document and an in-person walk-through of the procedure, answering any questions they have, and then they take notes and someone in the group is charged to polish my original document to cover the points that *they* think are important or non-obvious, ideally producing documentation suited to the audience.
It's not a perfect system, but it works a hell of a lot better than when I'd send out documents that would get filed to some doc repository and never be looked at again. It saves me the time and struggle of determining what is too much or too little information, and the people who need the information now own it. It took some lobbying on my part to get buy-in for doing this, but so far it seems to work pretty well. Also, I work in a pretty small shop which makes it easier to try things like this, but I see no reason why a similar workflow couldn't work in a larger shop.
Re:hire a Technical Writer (Score:4, Interesting)
You'll never succeed at this. The Army has a test for officers in one of their advanced programs. The officer is given a mission and has to write out a set of orders for that mission. Those orders are then given to a unit to carry out, with one addition: the soldiers are to do their best to fail to carry out the assigned mission without disobeying those orders (technicalities are perfectly fine as long as no order is actually disobeyed). If the soldiers manage to fail to carry out the mission, the officer fails the test.
I believe this test has a completely unblemished record: no officer has ever managed to pass it.
Sometimes the fault does lie with the person carrying out the instructions. Every set of instructions assumes some minimum level of skill and/or intelligence from the person carrying them out. If you don't think so, remember that even a character-by-character walk-through of logging on assumes that the person knows to turn the computer on first. The instructions on how to turn it on assume that the computer's plugged in or the person knows where the cord and plug are and how to plug them in, and that either the instructions are written for one particular revision of a particular model of computer or the person knows how to find the power switch on an unfamiliar computer. I happen to have personal experience with just how bad it can get. Helpdesk call from one of our midwest truck stops, reporting that their pumps weren't working. It finally got escalated to me as the developer who wrote the pump-control software, since nobody on helpdesk could figure out why the pumps weren't responding. I went through a few basic checks, then told the manager to go out to the pump island, gave him a walk-through of the internal self-test of the pump, and asked him to run the self-test and tell me what codes it reported. His response: "I can't go out there. A tornado came through and tore the islands up, and the main electrical line's laying on the ground out there sparking.". Turns out that, besides the pumps being physically damaged, the power was completely down and the computers were running on UPS batteries. One would think that any reasonable person would've connected a tornado tearing the pumps half off the islands with the pumps suddenly failing to work correctly, but apparently not.
Re:the easier to edit the better (Score:3, Interesting)
Re:And When Technical Procedures==Scientific Metho (Score:5, Interesting)
I worked very briefly at Dell, doing corporate tech support (hardware), and well not to brag but I was still #1 in the stats ranking a month after I quit until my averages rolled off... anyway I spent much of my off-call time trying to figure out a way to condense my smarts and experience into easy-to-follow instructions for my neighbours, who were often struggling with what I considered very basic problems.
To put things into perspective, my average call time was 5 minutes. That includes the occasional hour-long clusterfuck, which means the great majority of my calls were under 5 minutes. The top 10 techs had averages in the 15-20 minute range, and most everyone else was 45 minutes and up. Well now what was I doing differently, besides being the guy with the most hands-on PC experience ? I was being lazy, that's what! And I was committed to sharing my lazy ways with the rest of the crew.
During the exercise. I learned a few things:
1. It is damned hard to put into words things that are trivial (to you)
2. Logic, much like common sense, is a rarity these days
3. Most people fail to factor in the true cost of support time
So for #1, I had a non-guru friend take instruction, helping me dumb things down and bridge the gaps until we both agreed he had mastered the situation. He would express his frustration at my poor instructions, and ask a zillion questions until I had a grasp of my own internal thought process, of things that I did automagically.
For #2, I ran through a large number of scenarios and wrote down my own inner thoughts at each decision point. At the end, I trimmed these down to a rather large, multiple-entry flowchart. The neat thing is it covered both specific problems like "my hard drive is dead" and fuzzy symptoms like "my screen is blank". You would start at the edge, identifying the main symptoms and then you followed the paths until they crossed. At the very middle was the final solution "Replace entire system", the catch-all in case no other endpoint was satisfactory.
For #3, I dug up details on the approximate cost of support time. This included salary, utilities, and amortized equipment cost. Then I made a list of common support resolutions and their total cost (parts, labour, shipping). This allowed me to show how a long call can actually be more costly than replacing the computer in its entirety, especially if the extensive troubleshooting still leads to component replacement.
So in the end, we had verbose instructions, a troubleshooting cheat-sheet, and a cost guide. It may have enabled incompetent techs to perform tolerably, but more importantly it gave everyone the tools to learn very quickly. After a week or two with these troubleshooting aids, many people had the common issues memorized and had developed strategies of their own.
That's what I consider a successful transfer of knowledge. Knowledge is much more than just static information, it's also the process to create more information.
Re:And When Technical Procedures==Scientific Metho (Score:2, Interesting)
I used to be in the troubleshoot everything camp, now I am firmly in the copy the user's crap and re-image camp.
I also found this PC boot failure flowchart to be of immense help.
http://www.fonerbooks.com/poster.pdf [fonerbooks.com]
Beyond that my procedures stuff is pretty high level, stating what is required and letting the operations folks implement it how they like.
The nitty gritty stuff I have had to write has been pretty specific i.e. DISA compliant CentOS box running Snort, and that's because the DISA and NSA docs for Linux aren't current to RHEL 5.2 and the guy who's Snort install doc we were using died.
http://www.internetsecurityguru.com/ [internetsecurityguru.com]
However I am thinking a Wiki would be sufficient for most folks needs, I like Mediawiki, it was one of three packages that came with my webhost, and it has been easy to use.
I work on the teach a man to fish theory most of the time, and ask my analysts and other coworkers to come to me after Google not before. But as a rule of Thumb if you had to find the answer to a non trivial problem through Google it might be a good idea to copy and paste it into the internal wiki rather than rely on you remembering your search terms, you being at work, and your internet connection being up the next time it breaks.
Can't be done (Score:4, Interesting)
Not the way you're thinking. Where I work we use a wiki (mediawiki) and it works as well as any system I've seen, probably better than most. Now for the "can't be done" part:
Tasks like building a server, installing some software, connecting cables, etc., can and should be documented. Troubleshooting however cannot. Troubleshooting takes profound understanding of all the parts surrounding an issue and how they inter-operate; the ability to think long, hard and meaningfully; the ability to devise experiments, execute them, collect and analyze data and arrive at meaningful conclusions based on all that -- the scientific method, in other words. This is the kind of job that humans do better than machines and it is proverbially difficult to even understand, let alone document -- witness the "expert systems" of 20 years ago.
The big problem is that all non-technically-inclined people, including most people in management positions, think what we do is something mechanical akin to what plumbers do -- no offense to plumbers, who do things I can't do. From a user's point of view, "I can't print my e-mail message" is not much different from "I can't wash my dishes". Locating a clogged pipe and clearing it does require some technical skills and can certainly be documented. The e-mail printing problem however is several orders of magnitude more complex than that and, although it's probably possible to write a flowchart to document any possible thing that could be wrong in this scenario, it would be ludicrously huge, silly really. And that's talking only about day-to-day. If I'm really good at what I do I will design and implement systems so that errors of that sort are minimized. In other words, we're being asked to do the job of civil or hydraulic engineers on a plumber's salary.
There, of course, is the crux of the matter. Microsoft in particular, but the whole industry actually, are guilty of selling to managers the idea that this job can be done by drones, at all levels. Management is happy because drones don't cost much. Then, of course, drones are all they can get for what they offer.
What you want is knowledge, knowledge that can be provided by some, but by no means all, not even most, of the people who work for you. And you're tired of seeing those few good ones go away, leaving mostly drones behind. The only advice I can give you is: do document what can be documented. Make it a priority and make everybody responsible and accountable for documentation. But do acknowledge the really knowledgeable people in your team, treat them and pay them just as you would any other kind of expert. I know you're probably not in a position of paying your tech guru at orthodontist levels, but try to treat her at least as well as you do accountants.
Re:hire a Technical Writer (Score:3, Interesting)
I restrained myself.
If it were just a power outage, I can see someone thinking that at least the pump control electronics would be on a UPS along with the computers and the pumps should work to some degree (although they'd just give a "Cannot authorize" or other error or would authorize but not actually pump fuel). But after having a tornado hit? Anything related to pumping gas other than "Has someone hit the emergency shutoff switch yet?" ought to be waaaaay far down on the list of things to worry about.