Writing Good Network Documentation?

_Hellfire_ asks: "As the Senior Technician at a small mobile computing service I have just recently wrapped up a rather large networking job. The network includes a Linux Primary Domain Controller, Cisco router, port redirection for VNC, tape backup solution etc. - all in all fairly complex and not the sort of thing that someone not already familiar could easily take over. I therefore want to write good, comprehensive documentation for other technicians in case I'm not specifically available. How have other Slashdotters tackled this issue? Where do you start? How do you make sure that everything is covered, particularly when you've spent the last three weeks looking at the same set of equipment?"

Docbook

The format of the documentation is also important. Give docbook [docbook.org] a try if you haven't already. Concentrate on the content.

Nothing you can do

No matter what you do, unless you spend the next 6 months writing a knowledge base on it, will you be able to document every little thing. Just try to make something easy enough for the typical *nix user to be able to re-build it if it gets totally fried and you were hit by a truck on the same day. Expect some calls for the time being until their admins get familiar more with the system.

Overview it

I overview the entire network, then overview the seperate systems, then go into intricate detail about everything i've done and why i've done it.

I do semi-detailed overviews to start because when I've needed documentation in the past, its been an emergency, and i've mostly needed it to know exactly what router X is connected to, basically how it is configured, and what it is used for, not the intricacies of the firmware pasted blatantly up front without any mention of what the router does.

do the documentation in HTML and hyperlink EVERYTHING that could possibly lead to a question or a footnote or any supplementary text. this is what hyperlinks are for but far too often i've seen network documentation handed to me as 5th generation photocopies that are near impossible to read. take your documentation to anyone that is covered by an NDA (if need be) and have them proofread your documentation. have him ask questions that he would ask if that network were handed over to him, and on his first day needed to find out why the WAN was down, but the wan router reported it as up.

then burn your documents on CD, and keep it close to your heart and any contractor that works for you.

that's what i do, but i'm paranoid. i've been the victim of poor or non-existant network documentation too many times to not be paranoid about it anymore.

well....

first, label everything clearly. 2nd, document what you've done. print documents, in case there's no computer around to read them on. screenshots or logs of what you typed and have done is better than handwritten notebooks. 3rd, train someone in the Why and How of a system and it's response and they'll figure out the What on their own, usually. also, check out this article at kuro5hin [kuro5hin.org] and then do what seems more useful.

OSI Model as a, um, model :-)

I've done more than a little bit of this in my time. I've learned that as long as it is well organized and understandable, you can never document too much. I prefer a layered approach that coincides with the different layers of the OSI model. I begin with drawings from layer 1 on up, basically from cable, ports, card/slot, box, etc. [Physical Layer] to DLCIs (Frame Relay), VPI/VCI combos (ATM) [Data Link Layer] to IP Addressing and routing information [Network and Transport Layers]. I create numerous drawings in Visio and write documentation in Word (yeah, I know, MS). When appropriate, I import my Visio drawings into the Word document(s). You can even structure you document using the layers of the OSI model as well. Hope this helps a bit.

Teach

Teach it to someone else and base the documentation on their notes, or even better have them write the documentation.

I'm not suggesting this as a lazy-man's solution or a way to to delegate all your work away. Your intimate knowledge of the installation makes some things seem obvious to you that would never occur to someone else. The person you're teaching it to is approaching it from the perspective of someone who knows nothing (beyond what's considered required knowledge for the position) and their notes will reflect that. You don't have to wrack your brain trying to figure out what the reader will need to know, your student will ask. If you really want it solid, have your student repeat the process with someone else.

The best documentation/procedures I've encountered were written this way, and it's become my technique of choice when faced with a similar task.

Pictures!

Especially in network documentation. Nothing beats back a snarl of wiring (even if you were neat, you know by the time someone else is going to have to work on it, it will be a mess) like some old-fashioned line drawings. And remember, only half the documentation is what you are writing down--the rest is in clear labeling of cables, equipment, and machines.

Aside from that, what some guy said above is perfect--a broad-sketch overview, a sort of narrative description of how everything works and fits together, and then detailed drill-downs in specific systems and sub-systems. No point in forcing someone to go through twenty paragraphs of detail when all they need to know is what Router A is supposed to connect to. The overview is often the most valuable part of the documentation.

I just use a basic outline format usually, but as long as there is a table of contents and some sort of logical progression to it, I don't think it probably matters too much.

Good documentation is what really distinguishes professionals from amateurs in this business.

As a matter of fact...

I've just written a short book on this. Documentation for SysAdmins [sage.org], SAGE/USENIX's 11th short-topic book, should be available shortly (probably at LISA 2003 [usenix.org].

Not much complicated

The network includes a Linux Primary Domain Controller, Cisco router, port redirection for VNC, tape backup solution etc. Thats complex? Almost any network would have Microsoft-based machines in it, therefore an Active Directory setup with some Linux servers in it, and a backup-tape system. The connection to the Internet is usually through a connection-sharing MS or Linux computer, or a linksys/netopia or cisco router. And I'm describing a small company. We have all that plus more, including ethernet, fddi and 802.11b, and a linux-based vpn and firewall using x509 certs. For changes, we just have some co-op or temp students we keep around with some experience who come in handy if someone major is replaced. We do have a large tree structure of documents as one big MS help file containing linked HTML pages and diagrams. It includes daily, weekly, monthly and annual procedures and license update requirements. The whole set is refreshed with new information continuously, and any joe IT guy gets a good understanding of the company's IT structure. The purpose of the document is not to explain technology, but act as a reference so it is based on bulleted points rather than paragraphs. It starts with the network technologies and OSes used, then goes on about the OS, patches, settings and major software installed on the 8-odd servers, and their purposes. An appendix lists all important contacts, licenses and security watchouts. This setting works with a company with one key IT person dealing with temp /co-op workers. If the organization involves IT managers and teams, then the documentation becomes horrendously complicated.

Best guess, and what I try and do

In my experience, there are 3 critical things to do:

1. Download all of the documentation locally, especially for all networking equipment. Keep all patches/firmware upgrades locally so that in the event of a failure you have everything you need onsite to solve the problem. I once had a router go down, and the only copy of the docs we knew of was online... Good thinking guys... :-)

2. Do an analysis of what would happen for each piece of equipment if it failed. A list of steps to detect the symptoms. Essentially, a list of things to help trouble shoot it. Do this, also for major configuration settings that could be wrong (Gateways, DNS, firewall rules, routing rules). So essentially list what you think it would act like if a switch broke. If the router broke. If the DNS server went down. What it would be like if the proxy server crashed. What would happen if the firewall settings got mis-configured. How they could tell the tape backups didn't work. Things like that.

3. Put in place a system that will help to pull the configuration off the machines. Creative use of SNMP, nmap and something like NetSaint (now Naigos) will help you pull the configuration off the running network to see if you can identify failures or changes in configuration. Document how the configuration should be, and check it using those tools with alerts sent out when something looks wrong.

Kirby

Bad starts lead to bad finishes

You should have started documenting before you started "actual work". This won't help you this time, but design the system on paper before you purchase stuff. That way, you'll have a plan set up that you can get other people up to speed with, as well as something that you simply edit when you decide that you spec'ed it wrong. Always start with documentation.

For the great unwashed

A true professional would already know all about documentation, but for those who don't, here [kuro5hin.org]are some [ntk.net] references [theregister.com] .

I'm not real sure...

but I read this on kuro5hin [kuro5hin.org] the other day.

one thing

Those that write great documentation tend to get let go by their employer during cutbacks. "We don't really need Mr. Doe, he has documented his setup for us. Mr. Smith over from accounting could do it, for 1/2 the salary."

IMO, YMMV.

Managers and documentation

There was this job I once had... it was a large project, and design work was mostly done and implementation had begun when I joined. I was astounded at the quality of the documentation: major design decisions and justifications, ideas that didn't work, things to look out for -- all documented. Not just that: it actually was updated as things changed! This inspired me to keep a diary of my first few weeks at work, which contained things like the organisation of the source code, the build system, environment variables necessary for developers -- the things that people already working on the project know but that might have been opaque to newcomers. Yes, we were proud of the system, and of the documentation.

Now we skip forward about a year, when it's time for annual reviews. One of the things that the moron manager dinged most people for was the quality of the documentation! I marched in and, in a high dudgeon, demanded to know the meaning of the outrage. The documentation was bad, he replied, because marketing and tech support couldn't figure out what the thing did. I tried to point out the difference between documentation for users and for programmers... I should have saved my breath.

There were other problems with this yahoo, of course, and most of the dev team quit soon after. The company went under a little after that.

I don't care what you do...

... just make sure you do - I just started a job at a charity spread across multiple sites in the city, and 4 months in, I'm still finding things I didn't know about.

Big things as well, like an online centre half the size of the main one we run, which no one thought to tell me about until their router died.

Most of these comments suck

I'm sad to see most of these comment suck. I hope this get through the signal to noise ratio.

Okay, here's the deal. There's not a whole lot of law out there to protect privacy. You could go after each Spamer, but that's like hearding cats.

That being said, they issued you a credit card. In order to do that they pulled your credit record. They were not authorized to do that. They are therefor in violation of the Fair Credit Reporting Act. Both you, and the credit reporting agentcy can collect money from the credit card company. Read the actual text of th FCRA for details and damage limits. Then send them a certified letter indicating you'll settle your claim for X dollars. If you can get a local lawyer to sign it and put it on their letter head even better. If anything you can be a real thorn in their side. Assuming you have a copy of the real application you've already won. You should write the credit reporting agentcies and indicate that company issued a card with out consent, and you want it removed from your credit record.

You may also consider reporting them to the FTC. Although with the current pro-business administration it may not go as far as it did pre 2000. If the FTC did decide to act it would cost them serious dollars as well.

I an not a lawyer, this was not legal advise.

Re:complex?

Just remember, no documentation == job security.

It also equals no advancement because you are too important in your current role. ie, your in a rut.

Re:HOWTO: write bad documentation that looks good

Would be nice if you pointed to the original article [kuro5hin.org].

nice comment, stolen straight from k5

nt

COMPLEX???

This complex network consists of one Linux PDC(you mean Samba), one Cisco router, one tape backup(possibly a second machine but, I doubt it)? You've got to be kidding, right?

Anyone that finds this network complex should not be allowed near the console! This is the most basic of internet connected network and identical setup can be found in hudreds of thousands of homes and small businesses.

None the less. Documentation is always a good idea. It should include a rundown of all services in use on the network including configuration files for these services, how the services are used and by whom and it should include a diagram of the physical network. If it actually starts to become complex in the future, using VLans and multiple frame-relay PVCs on a WAN then a logical diagram should also be included.

You know

You know, that explains a lot of the documention I've seen.