Ideas For a Great Control Room? 421
lewko writes "Our company is about to build a central monitoring facility and I'm looking for ideas/suggestions about the best hardware and the best way to make it comfortable for those manning a screen. It will be manned 24x7 and operators will be monitoring a variety of systems including security, network, fire, video and more. These will be observed via local multi-monitor workstations and a common videowall. This is going to be a massively expensive exercise and we only get one chance to get it right. The facility is in a secure windowless bunker and staff will generally be in there for many hours at a time. So we have to implement design elements which make it a 'happy' place. At the same time, it has to be ergonomically sound. Lastly, we will be showing it to our clients, so without undoing the above objectives, it would be nice if it was 'cool' (yet functional). Whilst Television doesn't transfer to real life always, think 'CTU' from 24."
Fake windows (Score:5, Insightful)
Good lighting (Score:5, Insightful)
Whatever you do, don't make it dark with blue-ish lighting, like on TV. That strains your eyes. Provide good lighting, and make sure the persons can sit or stand comfortably while watching the screens.
Some things you'll want to do (Score:3, Insightful)
Borrow some ideas from the utility control rooms I've been in. Everyone has and uses their own headsets, I might extend that to keyboards. Keeps people from passing the contagious thing of the week around a confined space when sharing monitoring stations.
USG Contractor? (Score:4, Insightful)
Environment factors (Score:4, Insightful)
Do ensure that it has nearby and adequate washroom facilities. Nothing worse than having to travel up a couple flights of stairs every time personnel has to take care of nature, know what I mean?
Adequate lighting, and ventilation / heating / air conditioning is also something to consider. Nothing worse then working in winter with cold fingers, let me tell you.
I've been NOC for just shy of 3 years now, and I can tell you the environment you work in plays a huge role in how comfortably you handle the workflow. Its nice to focus on the more technical bits such as equipment and infrastructure, monitors, etc, but do not forget that people have to comfortably be there for hours at a time. We do 12 hour shifts here, and the most important consideration would be the temperature and air quality, imho.
What (Score:1, Insightful)
Re:Environment factors (Score:1, Insightful)
Much harder than it looks in the movies (Score:5, Insightful)
So, a couple things to think about:
1. How much money do you really have? Lots of people think they have money, but run out when it comes to all the details.
2. Do you want flash, or functionality? The two are sometimes complementary, but often one trumps the other.
3. How many people will staff? What's the schedule? This helps you figure out workstation configurations.
4. Are you putting multiple tiers in the same room? This is "best practice" if you do it right.
5. Are you handling customer calls directly? Do you deal with customers?
Basically, you need to figure out a lot of goals. A true "global" NOC can cost $50M easily for a telecom or comparable organization.
I've been a big fan of Barco for large projectors, and their IP-based solution is quite powerful. Recently, I rolled out a "public safety" SOC (security operations center) with 8 SXGA+ rear-projection displays. The largest I've worked on was 40+ of that style of display. Your garden-variety projector isn't cut out to handle this kind of duty-cycle. They're not cheap, but they're designed to operate 24x7x365, and many models have multiple lamps, etc. so that you can service them while they're online. So here's a few more things to think about:
* What goes on the "big screen" has to be useful. It must be grokable in a very short period of time. If you can't look at it for 2 seconds and get a good idea of what's going on, it's too complicated.
* Multiple displays per operations person
* Operational "graphs" that show overall statistics that matter to the people working, not to management.
* Good task lighting. Good lighting period is everything. Pay a real designer to do this.
* Good seating. We have let operations people pick chairs that fit their needs. Expect to spend $800-1k/person on seating.
* Sound deadening/management. NOCs get loud, and managing the acoustics is important to make sure that people can "think" and they can interact with one another.
* Ticketing is everything. Look at systems that are available commercially and for free. Consider writing your own if needed. If the system is streamlined to your own business, it will always be an impediment to getting the job done, which means people won't use it. If they don't use it, lots of knowledge is lost and post-mortems are more difficult.
Also, a few things that seem superfluous, but ended up being critical in some places I've worked (not all these were at the same place):
* Virtualized desktops (think RDP, X11, etc.) so that people can move and maintain their setup :)
* Color-shifting lighting to compensate for normal rhythms of people on weird shifts. Turns out green is effective after lunch at helping people maintain focus. This isn't cheap, but it sure does have a big impact.
* Keep your customers OUT OF THE NOC. A glass wall into the NOC is fine, but actually letting them in is distracting, and depending, can come with legal issues around privacy, HIPAA, etc. Best to keep them at a distance.
* Before you let customers see the NOC, you warn people. We had a blinking lighting strip under the displays that was linked into the Crestron system so that you couldn't flip the LCD-glass for 10 seconds to give NOC operators a warning. You don't want customers seeing people picking their nose.
Finally, as nice as good facilities are, if you don't have the process and people, it's useless. People people people people. Good people create good processes. Promote from within, and develop a strategy to give people a career path. Otherwise, you'll burn people out, and get huge turnover. That sucks for everyone.
Not my area (Score:3, Insightful)
But my first somewhat obvious thought would be to build a very detailed list of what has to happen in the room, then use that to drive your design.
Positioning of people I would also imagine is quite important.. which groups of users need to communicate with each other.. who will be using the video wall.. who is going to be making the most noise (is someone going to be on the phone every 10 minutes.. if so a separate sound proof cage might be in order)
Things like white boards might also be a good idea. For all the high tech collaboration solutions out there, I've found nothing beats a whiteboard for figuring something out or just tracking status of a short term issue.
I'd also watch the cool factor stuff. A lot of the stuff that looks really neat on TV actually sucks in real life. Moody blue lighting for instance is depressing and hard on the eyes. Maybe you could have some kind of "holywood mode" switch or something for when people are being toured through.. though that is a little extreme.
Finally I'd say good quality monitors and the most comfortable chairs that the budget allows.
Great Hazards (Score:3, Insightful)
In such a situation the greatest hazard will probably be from your own company executives. Establish a great monitoring process and it is carved in granite that some executive will add tasks that take attention away from the primary task. Reports leap to mind.
The second hazard will be from employees that man the monitoring station. They can become disgruntled or even be paid to do foul deeds. Good encrypted backups kept off site may help as will a monitor that watches the people that man the station.
It may help if you disallow electronic gizmos of all types from being brought to work in that monitoring station. Also tools that could open a computer case and install a USB card make it clear that you need to have absolute control of all items brought into the room.
Beyond those factors have you considered Faraday shielding?
Treadmill Desks (Score:3, Insightful)
I'm serious. If people are going to be there for long periods of time, they should get a little exercise.
I'm not talking about running, just a leisurely ~1mph walk.
Standing and walking are probably the absolute best options for maintaining good ergonomics.
The office furniture company Steelcase makes one. [steelcase.com]
I'm not saying to ban chairs, keep them as backup and for people who physically can't stand for extended periods of time.
Multiple Computers and Synnergy for Videowall (Score:4, Insightful)
You can purchase some really high end equipment to manage multiple monitors on a videowall, but you shouldn't. Use standard PC level hardware (or lower end rackmount depending on space requirements) with no more than two display cards each. Drive all your monitors separately then tie them together with Synnergy. You can still administer them all from a single workstation, fairly seamlessly, but you don't have a single point of failure, and you've probably saved hundreds of dollars. The videowall systems can also run some light duty servers especially system monitoring. (I like Xymon over Nagios, but it depends on what you want to do with it.)
So far as the monitors themselves, purchase flat-panel HDTV's. They are likely to be cheaper than similarly sized monitors, and you won't want greater resolution than an HDTV can handle for a video wall anyway. This gives you the added benefit of being able to tie in training videos, or third shift entertainment on to one or more screens if needed. Also, if one of your videowall servers goes down right before clients come to view the installation, you can quickly switch those monitors over to CNN, CNBC or another relevant channel.
The workstation tables should be glass or some other surface that can support either dry erase or grease-pen writing. Being able do simple notes on your desk will reduce scratch paper usage and make maximum use of available areas. Glass cubicle walls will cut down on noise like a cubicle would, but does not give as much of the feeling of being in a box as standard cubicles. They allow unobstructed view of the video-wall and you can write on them with grease pens.
Have more workstations than you need, and do not tie people their workstations. If someone wants to claim one that is fine, but some people will really like being able to log off, walk across the room, and log back on. This will also allow you to bring in off-shift workers when shit hits the fan.
As a security measure, get a dot-matrix printer on your firewall. Feed tail -f /var/log/authlog directly to it. If anyone gets in that shouldn't they will NOT be able to erase their tracks.
Put in a breakroom or break area that still has a view of the common videowall. When your people are taking a break during downtime, they should still be able to see if it is suddenly no longer downtime.
For the love of God (and your staff) put in a drink fridge or soda fountain and a coffee pot.
Bathrooms, Janitorial Services & Housekeeping (Score:5, Insightful)
24x7 ops in a confined space that you want to actually work inside means caring about the two things overlooked by both IT and Management.
1) Human waste
Unless able/willing to get time outside the tank, people _are_ going to: snack,drink coffee & water, etc at their stations.
The bathroom (mens and womens) is going where? Outside the double tier biometric locked doors past the guard?
In a crisis or crunch the time use for bathrooms actually goes up as more coffee and crap food (fatty, sugary, glutten, etc) are ingested & a trip to the sewage system is required.
Also bathroom facilities need to be built with those things necessary for those with disabilities; with obestity / diabetes and those with really bad digestion (IE get a really good odour neutralizaing vent system) Also some sound damping is necessary. After a shift change would you want to hear 14 flushes in a row at your station?
If you want a real good idea of how human smells take over a confined space take Greyhound to anywhere on a trip longer than 8 hours.
After a few hours you'll be begging for fresh air & a decent restroom as well. You can figure out the max time people can hold it as would need to if they pass outside the security areas of your ops center.
2) Housekeeping & Janitorial
How are and just who will clean the bathrooms as well as the control room area?
The guys & gals making so little as opposed to the IT guys, but do the really important grunt work during the night to porter the bathroom with their cleaning carts and supplies.
Really, although the janitors have a key to everywhere they typically will block doors open. The cleaning team is so "trusted" it isn't questioned about bypass of security doors.
If Solid Snake could hide in a janitors cart not a cardboard box - he'd go anywhere.
Additionally these are also the people that are going to empty the waste bins, recycle bins, spritz down empty cubes/stations with disinfectant / de oderizers.
My former company had a policy: if techs are on a station that station will not be cleaned.
You have to have some sort of desk rotation to move out your personnel (that monitor now covered by a different station) so the area can be cleaned adequately.
If this is not done you deserve the thick stank that will descend upon your control room.
If your running really critical Ops: A HR policy on proper hygiene and showers should also a clause in the personnel contract. Everyone laughs until you have to term a tech for stinking to much.
As a bonus I'll add that my experience is that the AC design will never be adequate nor will the heating. It will be visited often by AC techs in the first 5 years until everyone gives up hope.
The failure is that AC is typically the retail mall design of a large scale dumping of cold dry air into a large volume of space and somebody's desk (hopefully not yours) is just under where this happens. Hot and cold spots are intractable in a large open floor plan arrangement.
Abnormal Situation Management (Score:3, Insightful)
The chemical processing industry have been working together for at least 15 years on related topics.
http://www.asmconsortium.net/deployment/guidelines/Pages/default.aspx [asmconsortium.net]
Might be something in there that help.
Specifically display standards and alarm rationalisation ("every alarm should have a unique action", if there is nothing the operator on shift can do about an alarm it should be journaled).
ZombieEngineer
Re:Good lighting (Score:2, Insightful)
I do complain about the flicker from PAL/NTSC TV. It can't watch PAL TV at all. It's like staring into a strobe for me. This could be because I only have peripheral vision to work with. LCDs are the best thing that ever happened to me.
Re:Careful What You Wish For (Score:3, Insightful)
Re:if you need to ask (Score:5, Insightful)
I am the OP. Thanks for your advice. You've saved us millions. No, billions.
Now that you've established I am clueless, go away and read some of the genuinely insightful comments on this thread and you'll see precisely why I asked Slashdot. It's called canvassing a broad range of opinions. Even the world's best consultants/designers would have only designed a handful of these personally. I don't want to make any expensive mistakes and I am quite happy to consider the opinion of a low-ranking staff member earning a few bucks an hour sitting in these places before I subject our low-ranking staff to the same mistakes.
And FYI, my consultants are reading this thread and loving it.
Re:Much harder than it looks in the movies (Score:3, Insightful)
4. Lockers outside the NOC for staff. Make them nice, tall and big, and nobody shares.
A locker-room with showers, connected to a workout room with treadmills and stuff, is almost a requirement for shift work. Nothing eliminates the "mid shift snoozies" like 5 minutes on the treadie. Or a semi-serious workout in place of lunch.
My advice is no webcams, maybe even no security cams. Guys don't care, but the women found it extremely creepy that some dirt bag was watching them stretch and bounce around, so they either complained or refused to use the facility. Now a glass wall, where they can see you, but you can see them, is a perfectly adequate non-creepy substitute if you're worried about people collapsing.
Speaking of cams, for all the time I've worked, women have complained about working 2nd or 3rd shift and parking in an empty pitch black parking lot. Come on guys, how much would it cost to light the freaking thing and stick a camera out there? Or have the architect design it so the security guard at his desk can directly look out the window at the "NOC" parking lot? I would think that from a discriminatory legal standpoint, intentionally designing a hostile work environment for women or an environment that makes the company liable for creating an unsafe work environment for women is probably a legal risk that is Oh So Cheap to fix that at the design phase and oh so expensive to settle, even out of court, when something very bad happens to one of your female employees in the parking lot...