Ask Slashdot: How To Evacuate a Network 331
First time accepted submitter gpowers writes "I am the IT Manager for Shambhala Mountain Center, near Red Feather Lakes, Colorado. We are in the pre-evacuation area for the High Park Fire. What is the best way to load 50+ workstations, 6 servers, IP phones, networking gear, printers and wireless equipment into a 17-foot U-Haul? We have limited packing supplies. We also need to spend as much time as possible working with the fire crew on fire risk mitigation."
Wrong Time To Ask This Question! (Score:4, Informative)
Not trying to be mean, but you should have already had a plan in place...this late in the game without a plan means you just have to go with asses and elbows (just get what you can while you can and forget trying to install a plan to do it!)
The good news is you can become an example for other IT people! Everyone should look at their disaster plans and make sure you have accounted/planned for all emergencies that may happen in your area.
Triage and Labels (Score:5, Informative)
Log off Slashdot before reading this :-)
If you've got labeling stuff around, use it (not fancy label makers, just the basic "Hello My Name Is" and a Sharpie.)
Grab the servers, grab the workstation bodies, grab the phones and anything else that's easily portable, and any backup media you've got. Unfortunately, rack-mounted equipment is usually harder to grab, but that's probably your most expensive and critical stuff. And it'll be your critical path, so start unbolting it first. All of that will fit, put it in first, braced as well as you can.
Monitors and keyboards are nice, but they're just money, not data. Grab a few of them, but leave the rest for last. If you have packing material left, great, but if not you'll just have some breakage. If you've got any CRTs, leave them, they're heavy.
have everything "ready to go" (Score:5, Informative)
thinking this through, your priority is to make it as quick as possible to get essential gear out the door. so, consider:
* having the servers and desktops *already* in easy-to-carry crates, with large handles on the outside and packing materials surrounding the machines.
* have the machines stacked off the ground so that people don't have to waste time bending down and possibly injuring themselves by jolting weight that's too much for them
* have all essential equipment nearest to the doors plural, prioritised by criticality
* yes doors plural: add an extra door next to the existing one (or replace the one door with easy-to-open double-doors with those pushable handles) so that at least two people side-by-side can get through at once, carrying the crates, and can just "barge through them" rather than having to twist the handles.
* make sure that the crates are stackable and sturdy but also light enough to carry!
* even consider having the machines already loaded onto 4-wheeled trollies and left on them, permanently.
* if time is _seriously_ critical, consider putting guillotines next to all cables (and test them) so that people don't have to waste time unplugging cables: just cut them and go - but only consider this if the guillotines are sharp enough and easy enough to operate, and only if it's seriously seriously critical to save seconds. don't put power cables through the guillotine though!
* consider getting convenient light-weight but sturdy cabinets made for all LCD monitors, with double doors that fold back 180 degrees out of sight, and a top (with a handle) that locks automatically when it's flipped over. have the LCD monitors mounted onto the cabinets with rubber bushes so that they don't need to be placed or positioned into the cabinets - just pull out the cables, shut the doors, slam the top over and pick it up by the handle: done.
* consider getting 12v powered LCD monitors instead of 240v/120v AC mains, so that the power cables can be guillotined rather than pulled.
* instead of guillotining, consider breaking all the tabs on the network and telephone cables (the ones that "click and lock") and affixing them *loosely* with gaffa tape to all devices (network hubs, machines etc.) - this way it will be possible to just pull (hard) and out pop the cables. or, if someone forgets, and gets to the end of the wire, they won't trip or be yanked backwards: the cable will just come out, clean.
* get 4-port hubs instead of 8, 16 or 24-port. 4 gaffa-taped cables are easier to pull out than 8, 16 or 24, and if one of the 4-port hubs is lost to a fire, so what, big deal. a 24-port hub however starts to get expensive.
* stop people from putting the bloody screws in the bloody cables - you know the ones: parallel ports, VGA cables, serial cables etc. the ones that are always bloody irritating when it comes to fixing or moving a machine and you find that the bloody VGA cable needs a bloody screwdriver to remove the damn thing. take the screws *OUT* of the cables; that way people can't go "oh look: screws - let's tighten them".
so - yeah. make it easy to just shift everything. have practice drills. set a deadline (say 1 minute) and see how much kit people can get out in that time, without damaging it.
oh - and you know how i suggested making it easy to shift everything? uh... make sure the insurance is up to date, and get good security. no point making it easy for *other people* to shift all that expensive gear, eh? oh. and sort out some off-site backups, eh? :) i use rsync; my friend uses backuppc (because he has a lot of machines). /peace
Leave the equipment, take the cannoli. (Score:4, Informative)
Walk out with your backups, and save the people first.
Re:Uh... (Score:5, Informative)
Offsite backup at the very least. Save your data and your people, and let the insurance company take care of the hardware. Loss of productivity is a problem, but you're going to have that anyway.
Mod parent AC up, please. Spending time on emptying buildings of hardware which should be insured anyhow is in the best case stupid, and could even be hazardous - if it holds up evacuating the area of humans as much as a minute, it's criminal sabotage of an evacuation.
You're not even supposed to grab your coat when a building is evacuated. Much less hardware.
Re:You don't. (Score:5, Informative)
This. The parent is already +5 Insightful, but really needs to be +500.
You don't evacuate a datacenter, you abandon it. Any other plan is a dozen different kinds of stupid.
At best you trigger a self-destruct (software or better yet hardware) to whip all data so scavengers don't get to it while you're fleeing.
Hardware can be replaced easily (insure it, duh). Lives and Data can not. So already have the data backed up offsite and let the lives flee as they can at the first sign of danger w/o being hindered by insanely stupid commandments like "save the copier!!!".
been there done that (Score:5, Informative)
Two years ago we go a call telling us the levies might not hold and if they burst (1 block away) we'd have 8 ft of water. We didn't really have a battle plan and we had a lot less to deal with than it sounds like you did, but we learned some lessons.
1) praise the lord we had good network documentation. Now is not the time to be writing down how the firewall and public and private LANs are plugged in together. Shut stuff down, and start placing network hardware in big plastic tubs. Have tubs handy for this, they nest nicely when not in use. Toss cables in a different tub. just wind them up best you can into loops and toss them in. there's probably not time for neatness, you can deal with that later. TAKE THE DOCUMENTATION WITH YOU. You'll feel mighty silly if that's left pinned on the wall. or I assume you have an electronic copy you can print when you get offsite. Make sure any servers with complex cable attachments (like to phone banks or security systems) have labels on the connectors.
1b) got your phone system documented too? this is a whole 'nother can of worms that often is forgotten about. Does anyone have a diagram of where all those punched down wires go on each block? If you have phone switching hardware to pack, make sure the cables are labeled, they will all probably look the same with the giant connectors that attach to the blocks. "We'll just call Al, he does our phone stuff." Oh, you don't think Al is going to be BUSY helping everyone else that is returning? Nothing's as fun as a 2-3 day wait to get your phones back up and running huh?
2) Label ac adapters. You need to know which unit wants 12vdc and which has 24vac, you don't want to fry stuff when you are trying to reassemble. every pack should have the model of the unit it goes to written on it. Gear WILL get separated from its pack during the evac.
3) label staff's hardware. It's very annoying trying to figure out whose beige box is whose later. and they will probably fight over monitors and keyboards later. save yourself the headache. If you are already under the gun, run to the store and get a dozen rolls of masking tape and sharpies and have the staff label their equipment while you're packing things up, full initials or names, I bet you have duplicate first names you don't want to deal with later. Make sure you label the phones.
4) have a plan for things you can't easily move. the corp office was also forecast to get 8ft of water and they were on the WRONG side of the dike so it was more of a "when" than "if". they had a very expensive multifunction printer that the service people told them they could have a tech out to take it apart (so it fit out the door) in three days, which obviously was silly. They rushed in a bunch of cinder blocks and lifted it up and set it on them 8.5' up. (I have no idea how they lifted it) In retrospect, the building got 14" of water and totaled it, they SHOULD have killed power to the building and took a saws all to a wall. OR at least watertight wrapped it before lifting. I've seen this done with entire cars when faced with an incoming flood or hurricane. Even if it doesn't keep out the water 100%, at least it will keep out the mud, which you may be very grateful later. Got a plan for your big server room ups's? those can be quite large and heavy, and are often hardwired into the AC, are you able and qualified to unhook it? Maybe you should call in an electrician now and change that armored cable to a dryer type plug? Have a place you can move big stuff that can't be evac'd to where it will be at least more likely to survive. Think of flood, fire, and tornado/hurricane, there's probably not one single place that will work best in all three cases. Smoke damage can be very destructive, simply having something wrapped in mover's visqueen may prevent unnecessary loss that the fire missed but the smoke got. Do you have a plan for that rack that's bolted down or won't even fit through the door?
5) Document what's been left behind. A simple way to
Re:Uh... (Score:5, Informative)
You're not even supposed to grab your coat when a building is evacuated. Much less hardware.
That's when it's an emergency.
This is more like: "There'll be an emergency a couple of hours from now..."
Re:Uh... (Score:4, Informative)
Agree. You evacuate a network the same way you evacuate your building's other utilities (water, electrical, furniture) -- you don't. That's what insurance is for.
Your insurance company will pay to replace anything that's damaged by the fire. They probably won't replace anything that's damaged as you evacuate and re-occupy, or for the work needed to put it all back together. Yes, this is a "moral hazard" situation, but that's not your problem.
Re:Prioritize (Score:5, Informative)
They're in a pre-evacuation phase. Smoke on the horizon and all that. Not to worry, the local sheriff department will kick people out long before there is a problem. They've done this before. Like every year in the past decade.
Re:Uh... (Score:5, Informative)
Your insurance company will pay to replace anything that's damaged by the fire.
Insurance does not cover any damage that should have been prevented or was caused as a result of culpable negligence, when the org intentionally passed up a reasonable opportunity to mitigate or prevent the damage.
If you had an opportunity to mitigate or prevent the fire damage because there was sufficient warning, and you intentionally avoided mitigating the damage, that a reasonable person would have taken actions to prevent, then your reckless inaction likely means that the insurance company is not obligated to pay for the fire damage that resulted from your inaction.
Re:Uh... (Score:5, Informative)
Well, it depends on where you are. In the Los Angeles area, they have fires sprout up all over. When it's very dry and there's a good wind, those fires can move *very* fast. It is possible that when you go home one day, there wasn't a fire. The next morning when you show up, the area may have already been evacuated, or your building is a lovely pile of ash and molten computer parts.
When I moved away from LA, two days before we were to leave, a wildfire started several miles away. The day before we were to leave, we could see it on the next hill, about 1/2 mile away. We left early. The fire continued into the neighborhood after we left.
Disaster planning isn't suppose to be "we can take all the stuff out when the problem happens". It's suppose to be "Some spontaneous event happened, and we've lost all of our equipment." It should be dramatic enough to believe the building spontaneously collapsed, a meteor the size of a cit bus fell on it, or a tornado blew it away.
So, to address the summary, plan for the equipment to be a total loss, and that you'll have to restore from off-site backups.
What I'd do, if I happened to be there, would be to grab the servers, the C-level and Director level workstations (abandon the monitors, keyboards, and mice), and leave the rest. Hopefully everything is on the servers. It's not practical to consider moving 50 workstations, monitors, keyboards, and phones. I'd sure hope their insurance is up to date. We all know that top brass will pitch a fit if their computers are destroyed, which is the only reason for considering them.
There are probably at least a few departments who will need their files (accounting, HR, and sales). For most businesses, that'd fill up the 17' truck pretty quick by itself. That's a good time to talk to the boss about the paperless office concept, off-site backups, and a proper disaster plan.