Ask Slashdot: Designing a Telecom Configuration Center? 52
First time accepted submitter Big Jim Taters (1490261) writes "I have been tasked with helping move our config center from one location to our Headquarters. I have a small budget and no choice in location. I do, however, have an opportunity to design the space fresh (well, kinda.) What we will be configuring is routers, switches, firewalls, and other telecom related devices. What I cannot find is any "Best Practices" or "Lessons Learned" out there. So I ask you fine folks: What are some of the best and worst designs, practices, procedures, and work flows that you have encountered in sitting down to stage anywhere from 2 to 200 devices at once to get configured?"
Leave room for a sleeping bag (Score:3)
and don't forget the raised floor
Re: (Score:2)
Leave room for a sleeping bag
And place earplugs somewhere around that spot.
Another datacenter destined for failure (Score:5, Insightful)
Either you padded your resume or your boss is dumb. Either way, you're taking a job away from one of the many qualified engineers who could succeed in this task.
belongs to:serverfault.com (Score:1)
Wiring (Score:2)
Many places use uniform cable colors to indicate the purpose of each wire (red = DMZ, yellow = uplink, blue = intranet, green = servers).
But I find that makes it very hard to trace individual cables. Tracing one red cable among a bunch of of other red cables is madness.
Rather, I'd suggest explicitly using as many random different colors as you can.
Label the end of each cable where it plugs in with its purpose if its not abundantly clear, but otherwise, tracing a single red-wire through a bunch of multi-col
Re: (Score:3, Interesting)
Having done enough of this (both telco 25 to 600pr) and data networking) - I don't buy that logic. Having a cacophony of colors next to each other makes them visually more difficult to trace back.
I like the idea of separate colors for separate purposes. However, that requires that you purchase separate boxes of wire for each. Requiring that someone actually maintain that architecture... and, it makes it easier for someone to just cut the 'red' wires to kill a specific subset of your infrastructure. Bad
Re: (Score:1)
Sounds like you work for a wireless company, or something small and or new.
Color coded cables that are labeled please.
--
Hey Barney, which wire is going to the extranet, intranet, Lilo port?
What's that? They all look the same? Bummer.
Bringing in various colors is not difficult nor does it appreciably change your costs you racist!
Re: (Score:2)
> My vote - one color - WHITE. Easy to label w/ a sharpie.
Why do you want to label wires with a sharpie? If you need to do so you are doing something wrong. Just use preprinted labels with numbers and barcodes (to easly scan these numbers). Attach label to each wire end and then input it into configuration database. In such database you can add all information you need - functional connection, physical connection (as which device goes where), even that floor positioning system - you have no limits. You j
Re: (Score:1)
Why label them at all, Seriously? Step back and look at this from a big picture. I oversee several large datacenters and many smaller ones and even some where a floor of workers is home run back to the cores instead of a separate user switch and I find the overall practice of loosely Velcro'd cable bundles and looking for a the last few numbers of the MAC address or using CDP neighbor on the switch and endpoint is MUCH easier, just as quick, and 100% accurate every time compared constant time consuming met
Re: (Score:2)
If you have lots of money, buy PatchSee cables. If not, install your switches so you don't have to run your cables from the front to the back of the cabinet. As for the rest . . . I may be looking for a job, but not for free :)
Re: (Score:2)
I would rather put additional spare wires in connections between core units (racks?) just in case you need them. If one wire fails than switch to another one, mark that old one as failed and be over with it. You can trace it if you WANT to but you don't NEED to do that. It is good to have some spare backup wires.
Concise Checklists (Score:3)
List out all your common changes, and produce a checklist template for implementation. This checklist should NOT be page after page of screenshots that nobody but the greenest admin will ever read. They should be concise, and contain just enough information to have all the implementation data, and jog the memory of the admin as to precisely which steps need to be done.
On the template, you should record all the data you can possibly need to implement the change. If you could not fill out the checklist, and then hand it to another admin for implementation, the checklist isn't good enough.
So, that covers the change request part of the checklist.
In the actual implementation part, record ALL the steps where there's a decision point. (As in, you don't need steps for "Remote in to admin console, Login to Switch Config App, Login to Switch, Enter Config mode, enter VLAN subsystem, etc.) "Add VLANs to switch, using information listed above" is fine. Make sure the checklist includes updating whatever documentation you have.
Each line on the checklist should contain the date/time the step was completed. (If the admin just has to put an "X" there, guaranteed they'll ignore the checklist and just put in the "X"'s at the end.
Make the filled-out checklist itself part of the change record. Your change records should be complete enough that you should, in theory, be able to take the pre-change-system config, execute the tickets one after another, and end up with the same final config.
Lastly, do NOT require mgmt. approval for routine changes. Your checklist should already cover giving the appropriate people warning of the change. If you require mgmt. approval (or a change control board) for the most trivial changes, it quickly becomes rubber-stamping, which is even worse than wasting everybody's time. Save the change review process for changes not covered by the checklist.
Re: (Score:2)
Keep your text config files in git, and use gitlab or something similar to be able to browse and comment on, and link to from wiki documentation, all the changes.
Organization is Key (Score:1)
Simple things, such as one pile (table/area) for unstaged devices; one table for staged devices and one for problem devices.
If you are required to document the process (most likely you are) - serial numbers, user assignments, etc - then the help of a USB bar code scanner and a spreadsheet can really help. If you don't have a bar code scanner then a good magnifying glass may prove itself quite useful.
A mobile internet
Start from scratch? (Score:5, Interesting)
You're seriously starting from scratch? Oh man... wish I could do that.
Lets see... #1 thing... when I started way-back-when, we had this giant display board that would show alarms on equipment. It looked neat, and made us look fancy. Don't do that... It was useless to those of us fixing stuff. The only people that looked at it were executives. Every hour or two some VP would come running over "Why is NewYork blinking red?!?!?!" "Because the gateways is down." "Is that bad?!?!" yada yada
Setup a wiki... make sure everyone has permissions to edit it. Make sure you have procedures in place for how and when to edit it.
I don't know what equipment you're using, but if you have a choice, try and go to a monoculture of one manufacturer if you can. We had a huge mess of every type of device you could imagine from every company on earth. Over in California we had 3 Juniper routers. Every time something "strange" happened with those we had to call the "Juniper guy" You should all be experts in everything you have and the easiest way to do that is to go with one vendor. It makes it easier to hire for. "We're a Cisco shop" or "We're a Juniper shop" whatever... That's up to you and how talented the people your managements willing to pay for. If they only will hire people green out of techschool like my old boss was, then this is likely a good idea.
Setup a centralized on-call list that charts whos responsible for what. One of the worst things that can happen during an outage is that you're screwing around calling Bob to get Toms number, because he has to change the firewall to let Tim into the device he needs to fix. This goes all the way down to your facilities people. I had an outage once that we couldn't address because the basement of the building was flooded and no-one knew who to call to get the pipes fixed.
Is everything you do software? Or will you be handing the equipment as well? Everything I did was via-remote. I rarely actually touched equipment, we had field techs for that. But my buddy that does the same job I did, now has a job where he actually unboxes the equipment and installs it personally. If you're going to do that I'd recomend a good printer, cable labeling templates... practice using both. TOOLS! Specifically "Easyouts" for stripped screws. A Small hammer. Hand-screw-drivers. Wire snips, etc...
Along those same lines, a benchtop with every type of OS that will pass through your stuff. We had Every version of Windows, MACs, Linux, etc... usually these were just to prove the vendor wrong... We'd submit a ticket for a bug and they'd say "Oh, that only happens with a MAC!!!" so we'd test it out "No it doesn't" etc...
I'm not sure if I got overly basic with you there, but that's how we did it.
For actually configuring devices, we'd have one of the leads write a script and then foist it off on "the new guy" to put it on every device. It's been about 5yrs since I've done a config though so things might not be as simple with all the modern GUI stuff.
Re: (Score:1)
Make sure that the door to the room can be locked (key, combination, access card, ...). If possible, have a "man trap" or outer vestibule. Make sure that the room has UPS otherwise plan UPS in the design (each rack, or a individual rack per row, or a flywheel, or ...). Make sure that the facility has a backup generator and that the systems are part of that circuit. Make sure to account for cooling efficiency (a hot-aisle/cold-aisle setup, or dedicated chilled water air conditioning on each row, or ...). Env
is this both remote and local config? (Score:2)
because they are way different.
remote config, you just need secure telnet and good management pipes, on top of the normal basic gizmo office (overpowered PCs, hopefully dual screens, redundant power, etc.)
local config, you will also need an open half-rack on casters at each workstation area, good AC and "battery" power, I like the idea of a cube terminal server 8 or more ports for you to get into the craft ports and push scripts, then localize afterwards. or a localization generator which saves all the con
Easy (Score:2)
I do that professionally... (Score:4, Interesting)
and I like to get paid for my work. I expect most of my peers feel similar. So as unhelpful as you may find this, hire someone who's done it before, and ask them nicely to let you tag along and learn. Then you can become one of the professionals.
The above statement may sound condescending, but it's not meant that way. It takes years to learn the stuff you're asking, and differentiates the juniors from the seniors. Asking the seniors to train you for free isn't likely to be that well received by most of us.
Re: (Score:3)
I understand where you are coming from, but I have to disagree...publicly.
If we took that attitude, why bother coming here at all? I come to slashdot to learn from those who have expertise in a given field, and lend my expertise in return. It's how we grow; individually and as a species. What purpose do sites like Stackoverflow serve in your world? We should all be working to help each other, not protect our meager little slice of the pie. What you know isn't a secret. You and you alone have not figured out
Re:I do that professionally... (Score:4, Insightful)
Actually, it's not so much an "unwillingness to share" even though I understand how it comes across that way. If it were a simple question, such as one might find on stackoverflow, certainly, happy to help. But the breadth of the question means a huge amount of time is required to answer it in any sort of adequate fashion. Time is money, they say, and frankly I have more money than time. So it's more like "there's a limit to my generosity, after that you have to pay for my time."
Console servers... (Score:1)
Get yourself plenty of serial console servers. 48 port ones are normal, so maybe you need 5. Then you can have specific places where each one goes. People go in once place, equipment goes in another. There's nothing worse than working next to hardware.
Re: (Score:2)
It's not just having to work next to the noise, it's the lack of space to work - you're plugging your laptop in and it's precariously balanced on something and you need to reach for something else, etc. etc. then OOPS! You bump the laptop, and it jerks the wro
Re: (Score:2)
Just having a desk in a server room with very long video/kdb/mouse/serial cables to a screen etc on the desk can make a massive difference on the cheap. If it's only going to be used occasionally on things that are not available on a network at the time there's not so much call for a KVM box with multiple connections.
A big bench is also good for setting up that new noisy server
Organization (Score:1)
Organization (Score:1)
Some random thoughts (Score:3)
You are asking quite a broad question so I'll just throw in some random ideas since the topic is so broad you'll need to specify exactly what you have problems with and ask about it... and then iterate and ask again since it would be a long process.
1) Mount infrastructure systems like UPS and AC in advance and let them dry run so you can test if they behave properly f.e. ACs dont produce fog (I've dealt with such problem and it required changes from vendor) UPSs produce weird vibrations and so on. Just put the basic stuff to work as soon as possible so it you can overcome any gotchas it may cause.
2) Plan for expansion in advance - make proper room for additional AC and UPS units. Wire it accordingly. Put exhaust tunnels and everything needed just don't put the unit itself - this will make it much easier for you to expand if needed as you'll have all the work done already. Also add all racks you'll probably need later and wire them. Just don't use them - leave as spare. Racks and wires are cheap - labour is not cheap and it consumes time when needed.
3) Put additional canals (just empty space) below AC units as they can leak fluid.
4) Invest into cable management system so you can label each end of each cable with barcode and have a database of this configuration so you can scan every end of every cable and lookup what is its function where and to what it is connected etc. Don't bother with manually labeling each cable since it is tedious and probably will change in time. Just use generic barcodes attached to ends of wires. MAINTAIN this database as it is the most important stuff. Every single change must be noted in database. Peroid. No exeptions. Also don't bother with color coding the cables - usualy you will end with a mess unless you have a simple network.
5) Invest in security system like webcams on entry doors, RFID IDs and so on. It is not as expensive and will give you audited information on who, when had physical access to the facility. Be strict and anal implementing this - in case of non compilance f.e. somebody is trying to get unauthorized access just raise the alarm to local security staff. It is better to have false positives here.
Build lab? (Score:2)
Your goals here are make it quick and easy to get stuff out of the box, configured, and back out the door as quickly and efficiently as possible.
Things I'd do to start:
If doing
Re: (Score:2)
> Separate but adjacent boxing/unboxing room with a sturdy table. And a sturdy
> cart to move equipment back and forth between them. You want to keep all the
> cardboard and styrofoam out of the equipment config area.
That is one great advance. DO NOT open equipment packaging in the data facility. Just do it somewhere else and bring it in unboxed. Uboxing in server room leads to loads of trash in the server room, dust and loads of other kind of garbage. Server room needs to be clean.
Clarifications and Answers (Score:1)
Re: (Score:1)
Re: (Score:1)
OpenDCIM (Score:1)
Be sure to properly document all of your network connections in a Data Center Infrastructure Management tool. I like OpenDCIM http://www.opendcim.org/downlo... [opendcim.org] because it's free and open source
Some Basic Todos (Score:2)
Small Budget.....eh...
Things to ponder on:
1) Primary and Secondary UPS/Generator to ensure good clean supply at all times. Depending on your power source and and stability, you may or may not need this. If you have a budget, go for it.
2) Primary and Secondary Temperature/Humidity Control to ensure a stable environment. Can get pretty hot in a Telecom room when AC is not working. Depending on your geo-location, you may or may not need this.
3) Raised Floor/Ceiling space to run cables. If budget allows, Cable
Tasked with moving config center .. (Score:1)
How many people has been allocated to the task and what is your budget and timeframe?