Ideas For a Great Control Room? 421
Posted
by
samzenpus
from the hunker-in-the-bunker dept.
from the hunker-in-the-bunker dept.
lewko writes "Our company is about to build a central monitoring facility and I'm looking for ideas/suggestions about the best hardware and the best way to make it comfortable for those manning a screen. It will be manned 24x7 and operators will be monitoring a variety of systems including security, network, fire, video and more. These will be observed via local multi-monitor workstations and a common videowall. This is going to be a massively expensive exercise and we only get one chance to get it right. The facility is in a secure windowless bunker and staff will generally be in there for many hours at a time. So we have to implement design elements which make it a 'happy' place. At the same time, it has to be ergonomically sound. Lastly, we will be showing it to our clients, so without undoing the above objectives, it would be nice if it was 'cool' (yet functional). Whilst Television doesn't transfer to real life always, think 'CTU' from 24."
Natural light (Score:5, Interesting)
Well, it sort of goes against 'being in a bunker', but if I was going to work somewhere for many hours, I'd like some natural light.
Of course it's still possible to achieve that using reflective tubing or such like, though it might still undo whatever it is you seek to achieve by being underground.
If it's not possible, I'd suggest paying lots of attention to lighting. And add some real plants too - they'll generate oxygen as well as making the environment seem less bunker like.
Water. (Score:1, Interesting)
Include a water cooler somewhere. If it's a secure facility, you don't want staff popping out every five minutes for water.
Dim lighting looks incredibly cool. It's also uncomfortable. Resist temptation.
Careful What You Wish For (Score:5, Interesting)
My company invested millions of dollars into a central monitoring facility, with a large video wall driven by Crestron equipment. The idea was the video wall could display news/weather alongside alarms/outages in real time, with geographic mapping capabilities. Workstations were quad displays on adjustable motorized desks which sat atop a raised platform for simple network runs. A large executive "war room" style conference room was built with a glass wall overlooking the platform and video wall believed to be useful in the event of some catastrophic failure. All other staff sat in cubicles surrounding the platform with glass cube walls anywhere that would otherwise obstruct the view of the platform/video wall. A secure mantrap was put in place to restrict access to the facility. Dedicated bathrooms were installed with showers in the monitoring area in case critical staff were quarantined for extended periods of time.
It was impressive when it was built, but within a couple years, the video wall has been dismantled and parts sold off due to its impracticality. The right software was never found to perform the type of "geographic" monitoring conceived, partly due to bureaucracy. Network redundancy was overlooked, which made the monitoring facility itself non-functional during an outage. The facility lacked appropriate backup generators and UPS to keep the facility running during a thunderstorm. The platform desks required too much real estate and allowed no room for growth, so they have been replaced by cubicles. The secure mantrap was an inconvenience for upper management, so the inner door was disabled, defeating the mantrap. The quad displays ironically obstructed the view of the video wall when it was still in place, and did not fit in the cubicles when they were installed, so these were reduced to 2. All critical staff were sent home to telecommute because they took up too much real-estate required for day-to-day operations, and it made more sense to not have critical staff in a single central location anyway.
The point is, don't get too caught up in building 'CTU' from 24. The right monitoring software platform makes all the difference, as does intelligent network redundancy, telephony and backup power.
Plan for fast depreciation (Score:5, Interesting)
I used to know someone who had worked for an alarm monitoring company. She said the chairs and other furniture were used 24 hours a day, yet the accountants were depreciating them like ordinary office furniture. As a result, the furniture was not replaced often enough and was falling apart and uncomfortable. Make sure to plan on fast depreciation for your furniture.
steveha
From my experience (Score:1, Interesting)
From my experience from working in the control room of a nuclear plant there are a couple of design elements that make life easier. First, any controls and monitors should be separated by an aisle where you have desks and computers. Additionally, if you have operators and supervisors in the same room, a tiered design is helpful. Think of it sort of like a semi-circle, with the controls and monitors at the lowest level, the operators desks and workstations at a higher level, and the supervisors in the back. This allows people to come into the control room to talk to the supervisors without bothering the operators, while the supervisors can still see what is going on. If a casualty happens then the operators will descend to the bottom aisle by the controls and the supervisors will descend to watch the operators and read the procedures. Management can then come in and observe from the level the supervisors previously were.
Second, it is useful to have mimic boards for complicated systems. This means you have diagrams connecting electrical busses and breakers or pumps and piping on the control panel itself.
Third, alarms need to be designed so that they can be quickly silenced, yet allow other alarms to come in. Preferably this should be over the controls where everyone can see.
Fourth, look for items where people can make mistakes. Color coded labels on your controls, monitors, etc., as well as logically positioning them can prevent mistakes. Don't just ask for suggestions. Do dry run evolutions and see where design changes can help. Build a mockup and run it with a crew.
Finally, don't screw up the HVAC design or soundproofing design. It will piss off your operators.
Re:Much harder than it looks in the movies (Score:4, Interesting)
Oh a few more thoughts:
1. Buy high end task furniture (Haworth, Herman-Miller, etc.) but buy it "used". It's 1/2 or less the price, and often you can get the used high-end stuff for less than commodity new.
2. Get a telephone system that doesn't suck. This is harder than you might think. Today, I'd build something with Asterix/VOIP integrated with a customer database to do some real-time CTI. In the past, I've used Aspect successfully as well. Cisco's VOIP gear is nice, but overpriced.
3. Everyone gets their own . Whether it be a headset, keyboard, etc. Trust me, it makes sense.
4. Lockers outside the NOC for staff. Make them nice, tall and big, and nobody shares.
5. Plan for actual breaks from operations. Nobody can stare at a computer screen that many hours and stay alert.
There's a million more details.
Re:Fake windows (Score:1, Interesting)
No microwaves, no fridges in your control area. Believe me, you don't want a small fire or anything to damage your main room. Make them in a separate room that is not too far to walk but a fire retardant wall and door in between and the sprinkler system separate from your control room as well. That way a fire breaks out and it just floods the "kitchen" and not your control room. Make sure there is a drain in the floor your kitchen as well.
Offtopic: But somebody had to post it (Score:1, Interesting)
http://royal.pingdom.com/2008/11/14/the-worlds-most-super-designed-data-center-fit-for-a-james-bond-villain/ [pingdom.com]
Coolest Datacentre ever. They have submarine engines for backup power!
Re:Natural light (Score:3, Interesting)
Re:The Red Button (Score:3, Interesting)
Noise (Score:3, Interesting)
Having worked in several such control rooms as a railway dispatcher, what made me mad was that the morons installed the computers under the desks (dozens, also every person working there had 6 monitors) with lots of ventilators and cold air coming from the AC in the room below. They could just as easily have put the computers in the room below, since they were in enclosed, locked metal boxes for security reasons, that way all the noise and cold air under the desk would have been avoided.
OTOH the noise-cancellation walls, floors and ceilings were OK.
If many different people sitting in the same seats, with varying weights and habits, the (expensive) seats were usually broken after 3-4 months.
One other thing, if they run a blocked, single app all the time, lefthanders need still a way to change the R/L mouse buttons, we didn't have that, it sucked for lefties.
Also, since the machines were totally blocked from accessing them, even changing a keyboard or mouse needed a certified tech, who had to be called from home during the night and holidays, that really sucked, how hard would it be to place connectors on the desk.
Great control room setup (Score:3, Interesting)
I used to work for a large insurance company in Chicago. The director charged with building our NOC in 2000 basically traveled throughout the country visiting other large IT organization's NOC's and took the best ideas and made them work for us - and it did resemble 24.
Take a large crescent shaped room with a 30' or more ceiling. The video wall was three different sections (this is important for separation of displays and multiple tools at the same time). The display units were high end rear projection systems that were each hooked up to computers that drove the display and were roughly 3'x5' each. Of course there's no seam or separation between the screens. Any group of screens can be used to display anything you want (1 screen, 2, 4, 6, all, etc). Pretty basic stuff nowadays, but it was great ten years ago. The left and right banks had three screens stacked on top of eachother, by either 4 or 5 wide. The center bank was 3 high by either 8 or 10 wide.
Three rows of crescent tables with low walls in front separating them, and minimal separation between workspaces - you want people in a NOC to work very closely with eachother, especially in case of an outage. Each station had two or three LCD screens mounted on articulating arms, but not to be stacked on top of eachother like those trading desks you see with 6 or 8 LCD screens at them. That would be too tall, and you couldn't look over the top to see the main video wall without standing. The room sat close to 50 people. Around the edges of the room are various cabinets, printers, personal storage for the three shifts of employees that work in the NOC, etc. Of course high end chairs are important as others have noted. Lighting is also equally important. You have to be very careful with making sure it is as close to natural lighting as possible. The lighting we used was recessed and inset so that no lightbulb shone directly out or down on the people - it made it less harsh, but still very bright in the room based on a good design. Wireless headsets are important, and also minimizing speakerphones and any other distracting noise.
Behind the rows of tables at the back of the crescent in the donut hole section if you will is an enclosed room large enough to sit 30 people comfortably with power, phones, and network connections to cover it. The walls facing the NOC are floor to ceiling glass, and it has connectivity to the videowall of the NOC so that displays from their can be sent to the meeting room as well. It has every high end normal conference room tool you could need - multiple video conferences, smartboard, integrated microphones and speakers, etc. Everything was hidden inside builtin cabinets made of high end wood. This main room is the situation room. During a large outage, 2nd and 3rd level staff will work out of the room in conjunction with the NOC teams. Directly upstairs from the situation room is another identical room, also with floor to ceiling glass walls looking out to the video wall of the NOC. This upper room was reserved for senior and executive management use during a large outage. Engineers and Executive management have different needs during an outage and require separate spaces and separate functions, although constant information does need to feed between the two. The upper room was more of the showpiece room. It had a motorized curtain that you could press a button on the wireless control panel to open and close. The entrance from the building going up to the second floor board room does not give anything away for what the NOC itself looked like, so once everybody was assembled in the room and the button was hit, it never failed to impress first time visitors. They would always leave their chairs at the conf table and walk right up to the glass wall to look down at the people working in the NOC and see what was displayed on the board.
It was an extremely impressive setup. I am now in sales and visit customer sites on a daily basis and I have yet to see something that even approaches what this
Re:if you need to ask (Score:3, Interesting)
Xybix. They make desks.
Everyone seems to talk about all the benefits of standing, vs. sitting, or more importantly the ability to do either one, but nobody ever bothers to propose something that does both at the discretion of the operator.
Xybix does both. Motorized desks that move up and down (desk surface and monitor platform independently), foot warmers (keeps people from fighting over the thermostat), air circulation, etc. I don't work for them, and I don't sell them, but I have worked at a few installations that have them and they're worth every penny (even if they are difficult to cable due to the moving-around nature of the thing). In my experience, they've been very able to meet custom demands, as well, and behave like a small build-to-order shop that happens to produce high-quality furniture that fits together properly as if it were one-size-fits-all (even though it's not).
If I could afford it, I'd be sitting at a Xybix desk right now.
Re:if you need to ask (Score:3, Interesting)
If I could afford it, I'd be sitting at a Xybix desk right now.
Not standing?
We've looked at motorized desks. The seem cute but having spoken to people who invested MASSIVE cash in them, their users seldom if ever adjust them. I asked their management for an opinion and they were adamant that they were a waste of money. Moreover, they've had a hard time maintaining a consistent look with various proprietary furniture that needs to be replaced over time and they suggested avoiding bespoke furniture wherever possible and keeping it simple.
I am however considering having a standing workstation or two to achieve a similar aim, if an operator wants to stand upright to work. They can all hot-desk, so this isn't a big deal.
Re:Bathrooms, Janitorial Services & Housekeepi (Score:3, Interesting)
2) Housekeeping & Janitorial
Don't only think what, but also think how. There's nothing more fun than trying to solve a critical time sensitive problem to the background music of a noisy vacuum cleaner. The control room at our refinery has a novel remote compressor with some air ducting that runs under the floor to vacuum ports at the wall. Rather than drag a vacuum around the cleaner will come in with essentially the tube and handle, and plug that tube straight into the wall.
The end result is the only part of the vacuum cleaner you hear is the person walking and the gentle sound of dust rolling up the pipe. While I have no hard stats on the results I'm sure this has saved not only the sanity of operators but also the lives of cleaning staff.
Re:Bathrooms, Janitorial Services & Housekeepi (Score:3, Interesting)
If Solid Snake could hide in a janitors cart not a cardboard box - he'd go anywhere.
In my unfortunately extensive experience with this over the decades, the problem is not so much James Bond sneaking in, its "trash bags" full of laptops and HDTVs sneaking out. Even in the "best cities". Everythings gotta be locked down, even/especially in a security theater environment. And all employees need at least one lockable drawer for their purse, cellphone, spare change, insulin syringes, whatever.
You really need to understand what security theater is before you plan your "cool security system". Good security can be annoying, so sometimes people do the security theater thing and install annoying things mistakenly thinking that must make it secure. All the biometric hand scanner, card reader, and DNA sampler will do is annoy the janitor as he hauls his bag-o-laptops out the door.
The worst thing you can do is contract out the janitor to a random collection of temporary illegals. Then anyone can/will just walk in, block the door open, empty the offices, and no one thinks it odd. At least if you have a direct employee, at least HR had a chance to screen them and at least one person on site might know whom they are full access to the entire facility.
pros and pros (Score:2, Interesting)
There is a lot good advice posted. Much depends on the operations that will take place in the room. Most control rooms have daily cycles consisting of the day shift with maintenance working on things and interacting with the operators, a swing shift where things quiet down, and a graveyard shift that consists of long hours of quiet and talk among the operators to pass the time. When a process upset or emergency occurs the operators must respond quickly, but with a well-run process upsets don't happen very often.
I have spent too much of my life in control rooms, paper industry and power plant. Control rooms existed before computers, consisting of expanses panels of controllers and switches. Since there were so many controls, at least half of them could only be reached by standing. The operators were trim and in good shape. They frequently spent their time sitting down because there was nothing to do, then immediately came to their feet when action was required. Humans function much better standing up. They think better and it is best for them physically, the two go hand in hand. Many control displays work by touch. Proper display design is an elite craft. Arrange the displays so that normal operator input occurs standing and dealing with the display at eye level. People will object that keyboard input is required and keyboards have to be horizontal. I don't know your processes, but the vast majority of keyboard interactions involve display selection, alarm interaction, and numerical entry (setpoints, etc.). Sometimes tags get typed, it isn't frequent or common and almost always results from bad display design. Anyway, numerical entry can easily be handled by a vertical keypad. It certainly doesn't need to be horizontal. Display selection should primarily be handled through the displays, using proper design. Alarm interaction needs its own small keypad beneath the numerical one. In the future, voice recognition will be used with the displays, but right now it would be a gimmick.
By the way, my company provides simulations for operator training. Contact me if you are interested. Start-ups, shutdowns, upsets, alarms, tags, etc. All the usual suspects.
Re:Environment factors (Score:3, Interesting)
To add to your kitchen comments make sure that all the outlets are at least waist high. Also (because the electrician won't care) I would put at the maximum two outlets per circuit and make sure your microwaves are on separate circuits. If possible have your kitchen circuit box separate from anything else. You won't believe the things people will want to buy for the kitchen:
Ice Tea Maker
Popcorn Popper
Crock pots
Their "special" espresso maker
Food processor
Juicer
Deep fryer
Hot plates
Toaster ovens
Ice cream makers
I've found that the kitchen is the biggest failure point electricity wise. The kitchen in a business (and the bathroom) are the places that feel most "normal" to people. They tend to treat them like the one at home.
Also, make sure your workstations have the electrical capability to handle:
Personal heaters
Electric blankets
Coffee warmers
All manner of fans, clocks, phone chargers, iPod chargers, and the weirdest thing you can think of sold on QVC or infommercials.
Yes, it seems silly and unprofessional but in the real world these things do happen and its the little creature comforts that keep people happy.
Re:Bathrooms, Janitorial Services & Housekeepi (Score:3, Interesting)
In my experience employers that directly or indirectly turn a blind eye, ie who hire against national/local labor policies, already have deep problems anyhow. I wouldn't work nor recommend anyone work for such businesses.
For legit businesses, most housekeeping staff I've known are decent hard working folks often looked down upon by the Management and IT guys if even recognized at all. Most won't steal nor want to be implicated because it costs them their job or the contract instantly. No investigation period.
I brought the fact they block open the doors from my experience of having to account for disabled door alarms and other bypasses to areas meant to contain sensitive customer info. Its a real problem solvable by having shredders & screen savers internally to the command center.
In my experience its the "entitled" IT that typically steal. What happened to the old parts locked up in some drawer from all that upgrading last month? Its now in somebody's home system or ebay'd without a trace.
As for low skill employee, contractor petty theft (phones, change, purses) that is an entirely different realm of security.
However being wiped out of laptops and HDTVs either in one fell swoop (stupid criminals who have to fence a big score) or trickling them out (smarter but still stupid criminals who open up an opportunity for surveillance) is not the issue.
The financial loss is not the IT management concern. Equipment should already in plan to be replaced and insurance covers real theft.
However explaining to management the downtime, that the laptops went out the door without encrypted partitions, that the serial numbers of equipment were never inventoried is a bigger theft -- IT guys ought to know this as stock and trade and if they were paid big money for no real work -- this is basically ripping off the stockholders/owners. IT & management took "big" money for not doing basic diligence in their work.
Anyhow most every large building has a door -- the smokers door -- that never latches and just needs a few tugs or leverage to pop open. Its just a matter of observation and turning doorknobs. I'm sure the building rent-a-cop does that right?
Those criminals that try to score used electronics probably already need rehab. Else want to fence them for cash for getting presents for the wife or girlfriends.
No criminal retires on the fortune accrued by stolen used electronics from a business.
If they were organized they would hijack a trailer headed to bestbuy or frys and live well for a few months or 1/2 a year.