Ask Slashdot: How To Evacuate a Network 331
First time accepted submitter gpowers writes "I am the IT Manager for Shambhala Mountain Center, near Red Feather Lakes, Colorado. We are in the pre-evacuation area for the High Park Fire. What is the best way to load 50+ workstations, 6 servers, IP phones, networking gear, printers and wireless equipment into a 17-foot U-Haul? We have limited packing supplies. We also need to spend as much time as possible working with the fire crew on fire risk mitigation."
Welll... (Score:5, Insightful)
Prioritize (Score:5, Insightful)
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
Re:Uh... (Score:4, Insightful)
Off Site Disaster Recovery and Fire Insurance?
Offsite backups (Score:2, Insightful)
Damn if I'd go in to work to remove hardware when a fire is threatening.
I'm not paid enough to risk my life. Period.
You don't. (Score:5, Insightful)
The "best" way to evacuate a data center is to already have off-site back-up for your data in place, drop a fresh copy to portable media, and walk out. The hardware should be insured. The life of your and your people (at least some of whom should probably be helping their families evacuate) are far more valuable than a few months of making your insurer pay for rented hardware until your new machines show up.
Emergency packing (Score:5, Insightful)
First, triage the equipment.
You likely do not have time to pull disks from systems, so pack computers and
external drives first. Get blankets to protect things. Blankets start at the bottom
to act like a shock absorber.
Things like networking gear and wireless stuff is irrelevant compared to the
computers, and probably lighter. If you CAN, sure, save all that stuff too.
But the data comes first. Don't forget backups.
If there are computers with really really important or sensitive stuff, put
those in someones car in the backseat, again with blankets. If I seem
blanket obsessed, it's because I've found them to be available quickly
either from individuals or stores. Yes, bubble wrap or sorbathane would
be better but you aren't likely to have that stuff lying around.
Re:Uh... (Score:5, Insightful)
Re:Prioritize (Score:4, Insightful)
I have to agree prioritizing is key. Fires are live creatures and can go from bad to disastrous in a flash. Get the data first which includes databases, file servers etc.
When it comes to packing if you have limited packing supplies focus on the most critical and hard to replace stuff first. If you don't have enough stuff to package every thing then at least make sure that nothing can move around or fall over in the truck. All most every thing is built tough enough to handle a trip down even a mountain road as long as you drive slow and stuff isn't falling over and rubbing against each other. Every thing can be a packing supply. Coats, boxes. blankets, carpet, string, rope, cables etc.
Good luck and if you feel up to it give us an update when you are done.
YES! Save only hard drives (Score:5, Insightful)
1. label all hard drives. Drive position, and server.
2. place hard drives in anti-static bags
3. pack drives in foam.
4. get drives far, far away.
Hard drives are both the most valuable, and the most fragile part. Do not load them in a stiff suspension vehicle like a truck, as this bounces the drives. Choose a soft-suspension normal car.
Next take servers and network gear. Desktops are a maybe, as are phones. Ignore printers.
Tape a piece of cardboard over the face of an LCD monitor to protect it from casual bumps.
Above all, no data is worth a human life. No heroics. You're not paid for heroics.
Re:Prioritize (Score:5, Insightful)
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
Quick-disconnect hard drives. Everything else can be replaced by insurance, but your data can't. With what you've got listed above, I could hike out with your company in my backpack. The other thing is, consider the health and safety in your disaster recovery plan -- you should not expect, nor ask, your employees to stay until the last possible moment packing in equipment. Equipment can be replaced... lives cannot. Nobody should ever risk their life for an inanimate object in a business environment.
The other thing is, you should have a disaster recovery plan that includes regular backups to an offsite facility. Any disaster plan should be able to cope with "and then a giant foot appeared above the building and squished it flat." Yours should be no different. It might not be a wild fire that threatens your servers... it could be a UPS that shorts out, or a tornado, flood, a failed fire suppression unit, or simple human incompetence (Yes, I've seen stupidity kill buildings).
Any plan that relies on people staying in danger to save your business unethical, immoral, and probably illegal. So save what you can reasonably and without risk take, in descending order of importance... but recognize that there may be situations in which the only solution is to exit the building at a dead run and not look back.
Re:Prioritize (Score:5, Insightful)
Right idea, missing a detail. Get your data and hard to replace equipment (e.g. custom orders, long lead time, no longer available) first. That may be servers, or just the HD's from them. After that, everything that is replaceable can be picked based upon it's value, size, ease of removal, and available space. If you have to take workstation HDs, try to get one of each model of workstation so you have at least one machine that you know will work with that HD. It's not critical, but it can save you some effort if the facility does burn. Most networking gear, phones, workstations, etc. are easily replaceable, don't mess with them until the more important stuff is out.
And most importantly, DO NOT WAIT until you receive the evac order, start packing at least 24 hours before an evac is likely. I don't care what management says about taking down the network early, your data and your lives are far more valuable than an extra day working.
Re:Uh... (Score:5, Insightful)
Wow, way to plan for disaster. You should already have had the systems transfered either to a clowd or to your remote site. If you do not you've failed.
So what this gentleman says is correct. If you've not labeled everything including cables and have detailed drawings of the installations wiring you've failed.
So what you do is get out your label maker and tools, shut it all down and label everything. Then pack it as best you can in the truck. You can expect 30-40 percent startup failure when you get them installed and attempt a startup.
You might just want to consider building your next IT center in a shipping container that can be detached and loaded on to a semi. Done properly your UPS and AC systems would keep them alive until you could get to an alternate location with power and network which you should already have contracted for in advance.
Re:Prioritize (Score:4, Insightful)
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
I agree. But I would prioritize slightly differently:
1. Make sure all non-critical staff are gone, and are well informed of what's going on -- where they can contact people for further information, etc. (People)
1a. Whlie you're talking to them, ask the managers of departments -- off the record, of course -- if there are any department specific, hidden fileservers that need evac. I haven't worked in an office yet that didn't have at least one.
2. Make sure all backups are offsite, preferably in a dry, fireproof safe someplace. Ideally this step happened years back, and you can roll your eyes at this one, but lets be honest -- it didn't and you can't. (Data)
3. Disconnect servers from their racks. Any data storage stuff in there takes priority. (More Data)
4. Rack mounted servers go next (Servers)
5. The rest of the server room as time allows (Networking gear)
Anything after this is probably stuff you can skip, assuming you have good fire insurance. If you don't, welp. Honestly, start thinking like a thief, prioritize things that are expensive:
Harddrives are good to try, but it's easier to just pull the towers. Aim for any high end workstations -- the secretary's machine probably shouldn't go (but be aware that they may not have followed your server file storage and there may be data on that workstation not on the server), but the guys back in marketing? Maybe that top of the line workstation with the 30" monitor may need a second look. As mentioned above, many companies will have unofficial servers hidden around or local backups of department specific stuff, make sure you ask around if you have time to see if there's a file cabinet that needs placed on a dolly.
In an absolute pinch, just use wire cutters to disconnect workstations and get them on a cart -- DVI and USB cables are cheap. Monitors are next up on the price list. Printers right afterwards.
If you do not anticipate fire actually taking out the buildling, it may be prudent to grab trash bags and cover monitors and towers with plastic instead. This will help keep any smoke or sprinkler systems from pouring on them and damaging things.
If you have a basement, or a fire proof safe, tossing stuff in it may save it if you are absolutely out of time.
Re:Prioritize efficiently. (Score:5, Insightful)
Re:Uh... (Score:5, Insightful)
You might just want to consider building your next IT center in a shipping container that can be detached and loaded on to a semi. Done properly your UPS and AC systems would keep them alive until you could get to an alternate location with power and network which you should already have contracted for in advance.
Nods, and seconded... motion to carry.
Move in a Flash (Score:2, Insightful)
As everyone has stated so far, WTF are you doing on Slashdot, when your in an emergency situation. Now, onto my part, WTF are you doing as a IT Manager, and you don't have a critical battle plan, How the hell did you get this job in the first place. Your previous background was not in IT somehow. /done.
First thing first, Grab the essentials. Since you stated 50+ workstations 6 Servers and random IT 'junk' then your most important priority is obviously the data. Whatever your company's business is, I presume you have six servers because the important stuff is done on these servers, and the workstations are really just end user stuff. Park the U-Haul close to the building and get ready to launch.
--1) Mark your machines. IP addresses, Hostnames, etc each machine should be labeled before disconnection. Your first hour can go quick, if you use programs like Nessus, or LanGuard to map your existing network infrastructure. Getting a logical map saved of everything connected is a wise thing to do. Once a logical map is made, and physically your equipment is labeled... Step 2.
--2) Unplug the important. So you have some networking equipment along with your processing equipment(computers/servers) You probably want everything that could be brought back, plugged backed in, and literally your back up and running. Well. Time to grab the essentials. Grab your routers, switches, the Modem will be helpful. Grab the servers, and THEN grab the workstations. Load them up. You have laser printers? Are these printers networked? If so, you may want to grab the most productive Printer you have, and take it with you.
--3) Segment the piles. Obviously people want to keep their 'junk' together, however in a emergency situation, the most important stuff needs to be cared for first. So the IP phones, are probably expendable. Maybe not thePBX device because those can cost upwards in the thousands, so if its located on the wall where it should be, you can probably disconnect it and take it with you. The phones themselves can be expensive as well, but not even close to the device that serves them.
--4) When your out of the way of harm, you probably want to setup temporary shop. So wherever you end up, you are only going to setup the marginal needs to ensure you can pull records/data. So a Hotel, U-Store-It, or even a different office building, your going to need to obtain some replacement temp cabling, and desks.
As you can tell, if you left behind anything during your grab, your going to be spending money to have replacements. Hope the insurance coverage wasn't botched, because this is where you learn your company can get back on its feet. Otherwise, if you lost your data, your dead anyways. At least you have your data.
I could care less about 'Cloud Computing", but it really is a blessing to have some sort of off-site backup procedure in place. While your prepping to move, you could have had your data backing up online/offsite. During the long process, you could have recorded serial numbers, taken photographs of each room (before) along with itemized lists of what was in each room (or associated with each workstation). If you did a good job, you probably now have sufficient records of all your assets ahead of time. But if you suck at your job, you are probably frantic right now, and making bad decisions like asking Slashdot for advice, when you have few hours left.
Now my example of an emergency situation that came up on me at a datacenter (10k workstation/server environment).
Flashflood warning popped up on me, and corporate was out of town. Our datacenter was setup with a T1, Satellite, and a backup (consumer based) Cable Modem services. With our primary switches and routers located in a locked room. When employees started panicking that water was entering the building from the east end(where the ditches and drainage areas were), I kne
Re:Triage and Labels (Score:5, Insightful)
dont unbolt the parts it's time wasted and it will make transporting the lot easily while in rack with a buggy .. .if the cat 5's are landed at punchhed patch panels
just unwire the rack and take it out as a whole
you may be able to remove the cabling in large chunks without causing too much damage if any.
computers ? same as above. just take the stations and put them in large bins..If you got an apple producer
of similar large produce cases you may be able to fit all computers in one box which again is handy because
it keeps things together and the screens in the second box . Dont waste time on kb's mice etc unless you have a
lot of time on your hands..
get the heck out and keep people safe is first
hardware comes last.
it's useless to dead people
ric.
Re:You don't. (Score:2, Insightful)
Well, it's obvious the poster here was handed the job of preparing a disaster recovery plan
No, go back and re-read the article. The guy works at a place near an actual forest fire that is burning right now. He is no longer in the "disaster planning" stages and is now in the "disaster plan execution" stage.
In that case, I'd say buy some quick-disconnect drive enclosures (the kind where you lift a lever and a harddrive is now dangling in your hand), write a formal letter of protest outlining exactly why you're not responsible for the company being wiped out, what mitigation steps you'd recommend with a proper budget, and keep a copy in a safety deposit box or some 'cloud' service far, far away from you...
Yeah, he's in the middle of a forest fire in the middle of a national forest deep in the wilderness in Colorado. It's a Sunday afternoon. Do you suggest that he drive down to the little mom-and-pop general store a half hour away and pick up some milk and eggs, and ask if maybe they have any extra rack-mounted quick-disconnect hard drive enclosures in the back?
Re:Like pictures are going to help on reconnects (Score:5, Insightful)
http://eddie.niese.net/images/Wiring_BAD.jpg [niese.net]
Re:Uh... (Score:4, Insightful)
Re:Uh... (Score:5, Insightful)
You're not even supposed to grab your coat when a building is evacuated. Much less hardware.
That's exactly correct. I've known people that literally got out naked (having been asleep), but they got out of a fire alive. If you know the fire is a risk and you can't replace the hardware for lack of insurance then the move should already be happening now...
"We moved all the stuff for no reason" beats "We lost everything because we waited" every time in the Thoughts of Tomorrow game.
People first, data second (Score:5, Insightful)
Save your data and your people
but not in that order.
Re:Prioritize (Score:5, Insightful)
They've done this before. Like every year in the past decade.
So the answer to the Ask Slashdot question should be: "Just refer to the emergency evacuation plan you drafted years ago, and have regularly updated in the meantime."
And if that isn't the answer, why the hell not?
Re:Uh... (Score:5, Insightful)
Mod parent and grandparent down, please, and tell both posters what doofuses they are. In hurricane country, do you think people are stupid for taking time to board up windows before leaving too? No. Spending time saving property when a natural disaster or like phenomenon is known to be on the way but remains hours or possibly days away from impacting you is, in fact, a DESIRABLE thing to do. Less property is destroyed. Less time and effort is spent replacing the property. Less time is spent filing insurance paperwork.
Perhaps you're in California and have earthquakes on the brain. It's quite different. This is not "get out or we're all going to die" situation. This is a "be ready to leave town if we tell you to" situation. Sure, if the guys in charge of evacuation tell you "leave IMMEDIATELY omgfire" then you do, and save the people, and throw away the property. But it doesn't always come to that.
Re:Uh... (Score:4, Insightful)
The Black Saturday fire front here in Oz was moving at 120kmh due to gale force winds coming off the desert, it did most of the damage in a single afternoon. Spot fires were being ignited 20km downwind from the main front which was creating its own local winds due to the massive column of hot smoke rising 15km into the atmosphere. The 3000degC heat from the fire front was leathal at a range of 200 meters, one (un)lucky survivor had his windsceen melt and drip onto his legs while he was taking cover under the dashboard wrapped in a heavy wool blanket. Another similar bushfire I wittnesed up close and personal was the Ash Wednesday fires, both fires continued to burn more or less harmlessly for weeks after the damage was done, at one stage the AW front streached unbroken from Canberra to Mallacoota (~500km)
Yes you may have a few hours warning if your lucky (ie: phone/power/repeaters are still working), but most of time the only warning is a lot of smoke and no bird noises, it makes the roads just as confusing as a smoke filled building, you have no idea where the fire front is or which way it's headed until a 100m high wall of fire runs over you at high speed. Admittedly the Black Saturday fire is an extreme example but a bushfire front moving at highway speeds is not uncommon in this country and I believe smaller fire fronts in California can also reach similar speeds when driven by strong desert winds or running up the side of the Hollywood hills. Footage I've seen of these looks very similar, I recall seeing a yellow timber mansion sitting on top of the ridge, it was being filmed at a distance by a helicopter crew, it's heavily smouldering walls literally exploded into flame before the front even got to them. Much like a piece of paper bursts into flame when held above a candle flame.
Of course as you imply the smart thing to do for any fire is to have a plan before the fire starts, If you invest some non-trivial thought, money, and effort into things like not putting your mansion on top of a natural blow tourch, water tanks, roof sprinklers, pumps, dug-outs, fire-proof windows/shutters, etc, you and your property can stay and fight the worst firestorm, it's a safe bet that at least you will come out the other side relatively unscathed. If you're like most people and your plan is to evacuate, then evacuate, don't fuck around loading a 17 foot trailer or looking for the cat. The servers can be replaced, the trailer is a fucking Albotross on a smoke filled road where you may have difficulty seeing past your hood ornament and may be directed to turn around by fire fighters, and the cat can cope with a bushfire much better than you can.
*You - Not 'you' the AC, rather 'you' the Darwin award candidate that posted the question. A final piece of advise from someone who has wittnessed several serious bushfires first hand, if you are already working with the local fire brigade then ask them for advise on an evacuation plan in the case of bushfire rather than slashdot. I'm certainly no expert but looking at the immediate topology around the center (in 2d via google maps) it looks like the property sits on a small hill that itself sits at the bottom of a large natural bowl in the mountains, it also has a creek or river on one edge. Maybe with their help it would be relatively simple project to set up and defend the property as an official refuge point for the area. Which, if I'm guessing right, would be the kind of thing the center would be eager to advertise to their customers and insurance broker.
Re:Uh... (Score:4, Insightful)
Can we change
"Save your data and your people"
to
"Save your PEOPLE and your data"
??