Required Practices for a Network Operations Center? 26
hayduke.com asks: "I've recently been assigned to a program that is designing a 'Network Operation Center (NOC)'. I started to look for books, online material and other sources to help define a baseline for the Services Level Agreement for our intended customers. Not having any customers yet we are trying to incorporate the design elements that will provide the best possible level of service to the largest number of customers. A search on my favorite search engine brings up a lot of articles that have companies boasting that they have been recognized for being 'Best Practice' leaders in their respective fields but there are no references as to what those practices are. As this will be a NOC (pro-active) as opposed to a Call Center (reactive), I would like to know what other people think that NOC should be at bare minimum or if there are 'standards' that all NOCs should be held to."
Something not mentioned much (Score:1)
You need to have... (Score:5, Funny)
2) Lots and lots and lots of blinking lights.
3) Biometric-based access, such as finger, palm and retinal scanners.
4) Big, ugly dude guarding the front door.
5) Hire Linux weeniers from slashdot to run it all for you.
Re:You need to have... (Score:2)
Give the people working there military sounding titles for wargames. For example, the guy who watches the WAN is the "Night Distributed Network Watch Commander". The guy who watches the mainframe consoles is the "Enterprise Systems Surveillance Officer". The manager on duty is the "Command Post Commander in Chief".
You also need a electronic map of the world, even if your company only operates in one city.
Re:You need to have... (Score:1)
Re:You need to have... (Score:1)
http://slashdot.org/article.pl?sid=02/11/01/232
.
Aeron Chairs! and other premium snake oil (Score:5, Funny)
I know we're just joking about these requirements, but they're scarily familar to three NOCs I've been involved with. I think there must be some unwritten ruleset that goes something along these lines:
1) 50% of NOC budget must be spent on funriture and flat-panel displays.
2) Trendy lighting in NOC must seriously interfere with trendy displays. (example: if room is equiped with halogen spot lighting, at least one non-movable light should be aimed at a projection screen).
3) NOC must be located in the most inconvenient area of the most inconvenient building.
4) Actual NOC computers must be running the latest, untested wiz-bang buggy software on the latest, untested wiz-band buggy hardware.
5) Half of the NOC staff must be completely unskilled, impersonal, and unwashed.
6) The other half othe NOC staff must be anal, uptight, and permanently pissed off.
Server Room layout is another story... but does match rule #2 quite well... the perfect server room is often located as far from the building's loading dock as possible. With a proper pallet jack, it should take at least 30 minutes to haul a crated SGI Origin or Sun Enterprise server from the loading dock to the server room. Smaller items should take no less than 15 minutes. Shaky ramps, cramped elevators, and narrow hallways are a plus.
Re:You need to have... (Score:1)
Dear Slashdot (Score:3, Insightful)
Yours,
L. Ardass
Good accounting! (Score:3, Insightful)
Warnings on the first day of the month (if day 1's use * 30 will exceed limit) would be appreciated.
Warnings on days with more than 50% more than normal days use deserve a warning of some sort. Each client should be able to configure their own warning levels.
If network badwidth limits are known in advance, warnings at 90% and 95% would be very useful.
Learn from the pros (Score:4, Insightful)
Host a box at Exodus or Level3 and have a read of the SLA they give you. Beter yet, just call them up and ask for a quote and a salesman to call, no need to spend any of your own money. It's probably copyrighted so you can't just use it for your own customers, but it'll give you an idea of where to start.
The Definitive Guide (Score:4, Funny)
Contract someone who knows what they are doing (Score:5, Informative)
If you are gonna provide an SLA, you want TECHNICAL advice and LEGAL advice. Most SLA's are actually toothless in real life. The lawyers give you enough outs that you will never have to pay up with most customers (a few have the talent to see through the crap and make changes to your contract to put teeth back in.) Even though lawyers are expensive, it pays for itself in the long run.
The advise on slashdot is going to be spotty at best, especially in the light that so many NOC's are run poorly. Without experience on hand, you will run into the SAME traps / problems that most NOC's with inexperienced leaders run into.
Well, here are a few things that you may need.
A TESTED disaster recovery plan for servers, network, power, and cooling.
A trouble ticket system customized for your needs usable by inside and out (internally generated tickets and customer generated tickets.)
A network monitoring / management system that tracks not only subsystem availability but performance and keeps a history.
A customer management system that can bring up EVERYTHING you need to know about a customer, their syetems, their people, notification proceedures, etc. (this is VERY non-trivial)
A change control system - what happened, who approved it, who did it, how long did it take, what did they do, how did they do it, when was it active, how do you revert, etc.
Sigh. Setting up a NOC correctly is one of the most difficult tasks in IT.
Re:Contract someone who knows what they are doing (Score:2)
Of all the posts so far, this is probably the best. If you're just starting out, I highly recommend a consultant with NOC experience. 800 pound gorillas like me are available for this. Send me email.
Re:Contract someone who knows what they are doing (Score:3, Insightful)
A huge part of disaster planning is organizational, and planning, and documentation of the business practices, which will also apply to you.
If you draw up a plan for every class of disaster you and your clients may have, then the lawyers will have something to work for.
Re:Contract someone who knows what they are doing (Score:2)
You're starting a NOC now? (Score:5, Funny)
The Art of NOC-fu (Score:2, Interesting)
Take it from those who have done it (Score:2, Informative)
Read it here [donelan.com].
A lot is experience (Score:1)
Well, here's my .02$ (Score:1)
- Don't use anything by CA
- Check out NetCool for event trapping
- If you have a development team, consider making your own app, since off the shelf stuff might be a hard fit, where you are always trying to customize it to meet your needs and it never quite solves all your problems...
-
Get good people. Seriously. (Score:3, Insightful)
The blinkenlights, CRM processes, trouble ticketing systems etc are all lovely, but the thing that makes a difference is the people. In one case, there were 2 network guys we spoke to - one was great, used his initiative to sort stuff out, never lied to us or tried to fob us off, and kept us in the loop with what was going on. The other guy was technically prob. better, but used all the company's processes to hide from us. He was reluctant to acknowledge problems, rarely responded to voice or email, and gave the impression we were not really important to him. They both worked to the same SLA, processes, standards, etc. One was good at his job, the other merely good at technology.
So, I would suggest that instead of spending a lot of time on SLAs, you spend time finding good people. Monitor your performance not by "uptime" (one of our suppliers was monitoring our site using the internal network. They got 99.999% uptime, while anyone outside the NOC got "server not found"), but by customer satisfaction - contact your customers once every 3-6 months and ask them to fill out a satisfaction survey. Deal with issues they raise. Treat a customer who leaves you like a company crisis. Encourage your people to think about outcomes, not processes.
Sure, you need to know how your network is performing, what your customer's uptime statistics are, and have the facilities we have come to expect (including blinkenlights). Just make sure you've also got some cat5, screwdrivers, and free drinks for your customers, and that you don't get carried away with all the fancy stuff.
NOC Howto (from someone who runs one) (Score:2, Interesting)
2. Consistent Documentation/Knowledge Transfer among your technicians.
3. Consistent equipment, stick with one vendor and develop excellent relations with that vendor. (Example, I stuck with dell on a big purchase, got my servers and 2 extra computers, a box of posters, shirts, and other dell junk and 16 switches for free just for "being cool")
4. Adaquate troubleshooting database. Find a help desk software suite that fits the nature of your user/clients needs. Make sure it is scalable, and intuitive. Having a web based self-help package is also recommended... The more your clients can help themselves, the less your phone will ring.
5. Document everything such as set up procedures to password databases. (don't post it on the web interface of your help desk software lol!)
6. Have security a priority on every direction your NOC has a presence.
7. Set up your equipment to talk to YOU. Most management software (openview, webtrends, help dest software, etc) can send you emails if not text messaging and whatnot... possabilities are endless, but its nice to know something is broke before your users/clients.
Thats it. I can go on and on.
from experience. (Score:3, Insightful)
2. Phone calls sound like a good idea to clients but they can be a pain in the butt.. especially since many clients have difficult accents. They also require a lot of a technician's time. Consider having no telephone support, or only for large clients.
3. You need a good customer management and billing systems.
4. Good inventory system, tie to billing system.
5. A web-based interface to your router(s) and switches is advised.. it is also advised to tie this into your other software, don't physically unplug suspended servers, just suspend them from the billing interface and have it automatically use SNMP to disable their port.
6. Make sure the person handling your routing needs is sharp, I've seen datacenters where the 'router god' was learning on the job (not a bad way to learn, just a bad way to run a NOC)
7. Make sure that you do proper cable management, keep the facilities in good order, working restrooms. I've seen datacenters which would make the homeless cry.
8. Server monitoring system. For the convience of your technicians, a projection or large lcd would be preferred.. one of those cheaper dot-matrix LED displays would be ok.
9. I've been places where I've had to answer telephone calls, answer tickets, lookup passwords, and monitor servers.. all from different webpages behind the LAN. INTEGRATE. The closer and easier something is to access, the more useful it becomes.
Btw, I'm currently writing management software for a NOC; although it is proprietary software belonging to the NOC and I don't think they have any plans to sell it.
software you need (Score:2)
2. big brother - problem notification
3. mrtg - bandwidth (and other stuff) monitoring
4. find, grep, awk, sed, cat, less - problem troubleshooting
links to these are on http://www.freshmeat.net