Ideas For a Great Control Room? 421
lewko writes "Our company is about to build a central monitoring facility and I'm looking for ideas/suggestions about the best hardware and the best way to make it comfortable for those manning a screen. It will be manned 24x7 and operators will be monitoring a variety of systems including security, network, fire, video and more. These will be observed via local multi-monitor workstations and a common videowall. This is going to be a massively expensive exercise and we only get one chance to get it right. The facility is in a secure windowless bunker and staff will generally be in there for many hours at a time. So we have to implement design elements which make it a 'happy' place. At the same time, it has to be ergonomically sound. Lastly, we will be showing it to our clients, so without undoing the above objectives, it would be nice if it was 'cool' (yet functional). Whilst Television doesn't transfer to real life always, think 'CTU' from 24."
Control rooms at CERN (Score:5, Informative)
Some experience (Score:2, Informative)
See http://www.tbcconsoles.com for a few ideas.
Chairs - Haven't found anything to beat the Herman Miller Aeron yet.
If you're placing PC's in the control room, beware of heating and cooling requirements. Most common mistake I've seen.
Second most common mistake, way over-spec'ing the AV system for the videowall. Keep it modest, in keeping with your actual requirements and conop, with room to grow.
I like the single tier monitor approach. Get monitors above other monitors makes for an ergonomically uncomfortable experience.
Do not, under any circumstances, cheap out on the monitors. Buy decent. Voice of experience here, cheap has the potential to kill your project dead
Its going to all be for show. (Score:4, Informative)
Lastly, we will be showing it to our clients
In other words its going to all be for show.
Big projected screens showing something management finds important to brag about but the employees will never glance at.
All individuality quashed, no pics of the family etc.
My advice, in all honesty, is to build two. One that actually works, and one that is a star trek mock up. Whenever they did marketing picture shows they hired college age models to "staff" our network management center anyway, so non-operational equipment is not exactly a problem for the models to pose with.
Re:My favorite NOC (Score:2, Informative)
Re:ergonomics (Score:3, Informative)
Re:Environment factors (Score:5, Informative)
I worked for a couple of months in doing NOC work, you pick up some basic must-haves really quickly in regards to comfort, I'd like to add them. Temperature and air quality is among the very top priorities for sure, here are a few other points.
Get the best and most comfortable chairs you can find and then get the ones that are even better. People will be sitting in these 24/7, you need the best you can get or they'll get worn down in mere months, both chairs and people. Full range of adjustability is essential, as is full back and lumbar support. Look at the seats in taxis, cop cars, any sort of car where people do long shifts, look at long-haul trucks if you can. Taxis where I live are almost uniformly Mercedes-Benz or Volvo, partly because of their excellent seat ergonomics.
Get people out of the chairs. I know this sounds silly considering you've just bought amazing chairs, but give people options. Exercise balls, kneeling chairs etc., let people mix it up so they aren't stuck in an ordinary sitting position for hours on end. Some basic exercise equipment is good too, doesn't have to be anything more fancy than some wall-mounted bars etc., just to let people stretch a bit.
Quick access to toilets and kitchen(ette). This type of work demands as little downtime as possible. Make sure the kitchen has at least two microwave ovens, depending on the number of people who'll be working there, a good meal is essential to getting through a night shift and even the slight inconvenience of having to wait a bit longer can be amplified by being sleepy and in a job where you deal with very stressful situations on a daily basis. Night shifts have been shown to have a correlation with heart disease and possibly cancer. Don't make people wait too for their food as well :-) The place where I worked had lots of people doing 9-5 jobs as well and the cafeteria always kept well-stocked vending machines with sandwiches and the hot dish of the day for the night shifters. It was highly appreciated.
Speaking of the kitchen, get really good coffee. No scratch that, get great coffee and reliable high-capacity coffee makers. These people will suck down black coffee like you wouldn't believe. Keep the fridge(s) well-stocked with ice-cold Coke (or whatever caffeinated soft drink they like) for the non-coffee drinkers.
I cannot stress this enough: Have fresh fruit available at all times. This was an absolutely life saver for me. You haven't had a tasty apple until you've bitten into a fresh, ripe Golden Delicious at 4:30 in the morning after a long nights stressful work. Keeps people from gorging on chips and other unhealthy snack foods as well.
Lastly, let people have their distractions. We used fancy multi-monitor setups where we could put just about any system we monitored on whichever monitor we wanted, very slick. I can't remember what they called it, but it seemed to work pretty well. My point is that the universal setup seemed to be one monitor for desktop stuff, one for TV (most people had 24/7 news on) and the rest for monitoring etc.. The TV feed was brilliant, during the downtime you could catch up on the news and so on, it helped you get through the boring parts of a night shift.
I'm sure I've forgotten a lot of stuff, but these things stood out in my mind as the most important.
"designer" 4200k, CRI=85+, LOTS, or (Score:1, Informative)
( the best fluorescent light you'll find, is actually a mixture:
50% designer/high-CRI Daylight, and
50% designer/high-ColorRenditionIndex WarmWhite, mixed, so
you get both the yellow-red end of the spectrum AND the blue-green end, fully, and people look healthy in it! )
or you will keep losing people ( long-term ) to S.A.D:
We're *evolved* to have about *60*x the light we get in our "offices",
and it produces significant changes in our long-term function.
If you want functional people in there, long-term, you're going to have to engage the biological mechanisms of health in 'em.
That includes the requirement for lots of light, lots of fresh air,
stick a Consumer Reports recommended eliptical machine WITH heart-rate monitor built-in in the break-space ( so it is possible for them to get the aerobic vitalization they need, without detouring in their off-hours to some membership/gym ),
iow CONTRIBUTE to the vital health of the people there,
and they'll be healthily contributing to your work, long-term.
To do otherwise, and then pretend surprise when people aren't vital long-term,
aren't healthy long-term,
or aren't remaining long-term,
isn't worthy of the word "intelligence".
Another trick: if you NEED people to be alert to what they're seeing, you're going to have to work-with the 90-minute internal cycle...
Make it so that people are alternating, on/off, so the ON-time is only 90 minutes long, or you'll have eyeballs without alertness on the screens.
Also, group the screens so they are in 3-wide groups, and similar stuff in the same row:
most can't maintain awareness of 4 things at once ( limit extends from salamanders through us, so it's fundamental ), so making it setup so that one is looking-at only 3 monitors at a time ( the 3-wide groups ), and one can go row by row down one's block of monitors, looking for out-of-line stuff...
makes it more likely that someone's going to notice the anomalies that occur.
Go for it, get it right, enjoy the results!
Look at people who do it well (Score:4, Informative)
I had a short stint with a fire department in a fairly large metro area. They handle emergency response for a few million people and have a pretty efficient set up for their war room.
It is fairly roomy so that you can get up and walk around.
desks are set up in pods with groups a people sitting facing towards the center of the hexagons they are on the perimeter of.
They have phone communications with headsets that allow them to move around
there are multiple monitors on each desk and large monitors on every wall that can be switched to show any desktop
The lighting and noise were somewhat subdued, but in no way dimly lit or overly comfortable (no high back chairs)
I do not remember the snack food situation, but there is a lot of security and it would be a pain to have to walk in and out all the time)
fwiw, I would save your company a ton of money and just use IP kvms and a software kvm management solution to tie together a bunch of desktops in a relatively open area together where the operators have some room to walk around, but are not overly distracted or lulled to sleep
ya know, massive operations centers are just soooo glass house IS anyway, totally 80's thinkin
My control room experience at fermilab (Score:3, Informative)
These are just my experience. I've worked in SSN maneuvering (reactor plant control) rooms, and at Fermilab, the Experimental Areas, D0, CDF control rooms and now days, the Main Control Room.
Keep the racks sealed, front and back as much as possible, to keep dust in the room from getting into the electronics. You might (or not) be amazed at how much dust a control room can collect over a year.
Include a fair amount of sound deadening foam. again, it's remarkable how loud pcs can get, way moreso if you have to add fan cooled crates and such.
Stay away from trackballs unless you can get *really* *awesome* trackballs, they tend to collect crap inside and are usu. kinda a PITA to clean.
Beware of the temptation to put in q00l tracklights, as they have a tendency to cause a lot of glare. Keep bookshelvs away from the consoles as much as possible.
Have your operators face AWAY from the hallway. Consider keycard access to keep distractions down. Spend the money on comfy high-backed chairs. Kitchen immediately adjacent to the control room. Bathroom nearby. Consoles should go UP, not out; it's easier to look up than turn your head and it _will_ make a difference.
I would not curve your workstations; skooching from one point to the next in a curved layout requires a unique trajectory for each endpoint; straight layouts are easier to run your chair along.
Here's mine http://www-visualmedia.fnal.gov/VMS_Site/gallery/stillphotos/2006/0000/06-0022-30D.hr.jpg [fnal.gov] kinda big, and i'm in the picture! Looks old and clunky but we manage.
HTH
Tape reels, and an ergonomics person (Score:2, Informative)
DOCUMENTATION. (Score:5, Informative)
What hardware you have doesn't matter much. Any run of the mill dual headed desktop will do what I need - a few browser windows, a dozen xterms, and some email up in the corner. Anything else is just for show.
What's really valuable when I'm on point is documentation. I need to know:
What does this server do?
What's the procedure to rebuild it?
Who built it last time, in case I hit trouble?
Who's the business owner?
How do I reach them? A cell number would help.
What's the escalation path when they're AWOL?
Where are the contact numbers for our transit providers?
What are their SLAs?
Where's a map of our network, in case I have to creatively reroute traffic in ways that OSPF won't?
Is it up to date?
What services are exposed in the DMZ?
Where's the ticket requesting this port be opened?
Are there supposed to be 100,000 different IPs connecting here, or just three?
Where is the password vault?
What's the procedure to update the password vault if I have to change one?
Being able to find these things quickly will make me a much happier sysadmin than any creature comforts, excluding caffeine.
If you want to get into creature comforts in a windowless bunker, make it lighting. I don't want it bright, but it should be well designed to cover the space well. Good warm triphosphor fluorescents with high frequency electronic ballasts are much much nicer than the old cold ones with 120Hz flicker from magnetic ballasts. Color rendering index matters. That makes a good base lighting for the workspace. Then get a few of those industrial HID grow lamps, and have them light a big picture of a forest scene covering an entire wall. Or actually grow plants under them, and don't pay too close attention to what else people plant when you're not looking. Careful not to make it too bright, but the sun-like spectrum will break up the monotony of the fluorescents. Add some bright halogen task lights for when you need to see something well, without having to flood the whole room.
Raise the ceiling as high as you can. Rip out the ceiling tiles. Suspend the lights on cables. Let the ductwork show. Paint it all black. I hate living in a box. Exposing all the HVAC and cabling breaks it up, and I actually like the minimalist industrial aesthetic. If you want a softer look, hang some tapestries up amongst the machinery.
This probably won't get read but... (Score:2, Informative)
I work in a NOC, 12hr rotating shifts, and we work 24/7/365 (and 366 on leap years :)). What I recommend is very comfortable fully adjustable high back chairs with a head rest.
As for monitors, you need to give them fully adjustable mounts, with nice high quality wide screen displays (Some may say standard aspect ratio, but seriously, widescreen would make my job SO much easier). 4 Monitors would be good too, 3 is good, 4 would be far better. Give them a powerful computer, and make sure they are kept up to date hardware wise, plus, if these people are technical, and you're giving them the keys to your castle, make sure they have admin rights on their own machines so they can fix minor issues without calling your IT Disservice Desk. Give them relatively unfiltered internet, night shifts get VERY boring. Make sure every telephone has a corded and cordless headset for comfortable use, holding a handset to the ear for long periods is hopeless, and wireless is good for short calls, but you need wired for anything long.
As for lighting, task lighting that the users can control is very important, as well as dimable overhead reflective lighting is important. As for desks, adjustable hight is important for comfortable use, and make them BIG. As for your video wall, make it a good size, and have it easily changeable if someone notices something important is missing, give them TV, hell, if you can, give them 2 'tvs', one for news, one for something else, and give them the ability to change the channels if they want to watch a hockey game at night, or if the olympics are on, etc, etc, etc.... Make sure the enviromental controls are good, and not constantly changing. Give them the ability to play music for the whole operation center, entertainment is BIG when working long shifts.
Give them a lounge to relax in before shift, or on lunch, or after shift. I think thats everything I can think of for now
Problems with these rooms (Score:1, Informative)
I was in charge of a small control room for the military. Obviously I won't talk about the security aspects of it, but I'll gladly tell you of some of the more practical aspects of it.
1. The biggest factor right off the bat would be the proper environmental control. Our room was a small room off of a larger main room. When the AC went out, even in the winter time the room became unpleasantly warm very quickly. A seperate AC until controlled in each room independently is important. Ventilation was also an issue. Too many adult males on a bad diet in close quarters. Use your imagination.
2. Cleanliness became an issue, since it was in a secure area staffed by a large number of people the levels of dust became an issue. Unless you have cleaning staff that is cleared to come in the area, your staff is going to have to have access to cleaning supplies and effective vacuums.
3. Sound levels were an issue. A number of desktops, plus network gear, plus radio and com gear resulted in a quite loud environment even when the equipment was not in use. Sound dampening on the walls and quiet computer systems are a must.
4. Lighting levels were a problem. The room had been converted and not purpose built. With no outside windows and normal height ceilings led to some really bad lighting and lowered the morale of the people. High/vaulted ceilings are must and the ability to change brightness level will go a long way to helping out people with cycle clues.
5. Break areas and bathrooms facilities will be a problem. If they are inside the secure areas then you run into cleaning again, if they are outside then it becomes a difficulty of getting people in and out of the area.
6. Multiple monitor setups are a must, but don't over do it on video walls they hardly ever get used. Pick up those cool robot desk arms for your monitors so maximize desk space.
Re:Natural light (Score:2, Informative)
You can get natural skylights now that use fibreoptics to transmit the light.
http://www.skydome.com.au/products_parans_home.htm [skydome.com.au]
1 major requirement (Score:2, Informative)
Re:Much harder than it looks in the movies (Score:3, Informative)
4. Are you putting multiple tiers in the same room? This is "best practice" if you do it right.
Horrible idea. I encourage all my competitors to implement that idea as fast as possible.
You ever actually work in a NOC? I have, for decades.
You ever watched "Office Space" with the scene where the guy is utterly unable to concentrate because his cube is next to the receptionists cube "Welcome to innitech, please hold ... Welcome to innitech, please hold .. Welcome to innitech, please hold". Horrible awful idea. Good if you are my competitor, or if you intentionally are trying to destroy your own productivity so as to heroically rescue it later. Otherwise a pretty bad idea.
Promote from within, and develop a strategy to give people a career path.
Now we're in daydream mode, that would be the first NOC I've ever even heard of with that.
Do you want flash, or functionality? The two are sometimes complementary,
Sometimes? Sometimes? I'm not trying to be excessively harsh, but only someone with no NOC experience beyond watching TV could make that major and fundamental of an oversight.
It must be grokable in a very short period of time. If you can't look at it for 2 seconds and get a good idea of what's going on, it's too complicated.
Again, newbie mode here. Even 1st level retail customer service takes longer to figure out what problem is. When you REALLY need the big screen, as opposed to using it as a fashion statement, is when the MPLD network routing protocol is not converging after three hours and a team of ten local engineers, three field techs, and half the company at Cisco can't figure out why. Simple enough for the drooling salespeople to explain to customers whom know nothing about NOCs using short one syllable words and simple sentence structure is a marketing requirement not a technical / operational requirement.
Good task lighting. Good lighting period is everything. Pay a real designer to do this.
Personalize the chairs but not the lighting? Again newbie error. Best NOCs I've seen over a couple decades go for dim lighting to keep reflections / glare off the screens. Also a good excuse for no windows.
I see you forgot temperature management? Worst design I ever worked in had stadium seating, and either the HVAC blew air in your eyes which makes them water like hell after a couple hours, or we wore sunglasses which gave marketing/sales an absolute cow, or didn't blow air around so people at the lower seats were freezing while the top seats were sweating. Which is an epic fail, but not quite as bad as the place I worked at that had such good mantraps and powerful UPS/generators that when the power failed, equipment started failing due to lack of air conditioning within 30 minutes and people started failing (fainting, overheated) within an hour. Almost as bad as the "24 hour" NOC I experienced that took months to get the HVAC guys to turn on the ventilation during 2nd and 3rd shift (gotta be green, ya know). Then there's the place I set the thermostat to 73 and pried the dial off and dared anyone to make me reinstall it (and no one did) because clowns would set it to 85 when they personally felt cool and 60 when they personally felt warm, and the rest of us suffered. There is always at least one clown like that in any group of more than 5 people, so frankly you need to eliminate thermostats, or mount them but don't hook them up so clowns can work out their frustrations by adjusting a thermostat that does nothing.