Ask Slashdot: How To Evacuate a Network 331
First time accepted submitter gpowers writes "I am the IT Manager for Shambhala Mountain Center, near Red Feather Lakes, Colorado. We are in the pre-evacuation area for the High Park Fire. What is the best way to load 50+ workstations, 6 servers, IP phones, networking gear, printers and wireless equipment into a 17-foot U-Haul? We have limited packing supplies. We also need to spend as much time as possible working with the fire crew on fire risk mitigation."
Uh... (Score:5, Funny)
Re:Uh... (Score:4, Insightful)
Off Site Disaster Recovery and Fire Insurance?
Re:Uh... (Score:5, Insightful)
Re:Uh... (Score:5, Informative)
Offsite backup at the very least. Save your data and your people, and let the insurance company take care of the hardware. Loss of productivity is a problem, but you're going to have that anyway.
Mod parent AC up, please. Spending time on emptying buildings of hardware which should be insured anyhow is in the best case stupid, and could even be hazardous - if it holds up evacuating the area of humans as much as a minute, it's criminal sabotage of an evacuation.
You're not even supposed to grab your coat when a building is evacuated. Much less hardware.
Re:Uh... (Score:5, Informative)
You're not even supposed to grab your coat when a building is evacuated. Much less hardware.
That's when it's an emergency.
This is more like: "There'll be an emergency a couple of hours from now..."
Re:Uh... (Score:5, Funny)
Re: (Score:3)
"There'll be an emergency a couple of hours from now..."
Thank god for that, it gives me just enough time to ask slashdot what to do with 50 servers and a 17' trailer.
Re:Uh... (Score:5, Insightful)
You're not even supposed to grab your coat when a building is evacuated. Much less hardware.
That's exactly correct. I've known people that literally got out naked (having been asleep), but they got out of a fire alive. If you know the fire is a risk and you can't replace the hardware for lack of insurance then the move should already be happening now...
"We moved all the stuff for no reason" beats "We lost everything because we waited" every time in the Thoughts of Tomorrow game.
Re:Uh... (Score:5, Informative)
Well, it depends on where you are. In the Los Angeles area, they have fires sprout up all over. When it's very dry and there's a good wind, those fires can move *very* fast. It is possible that when you go home one day, there wasn't a fire. The next morning when you show up, the area may have already been evacuated, or your building is a lovely pile of ash and molten computer parts.
When I moved away from LA, two days before we were to leave, a wildfire started several miles away. The day before we were to leave, we could see it on the next hill, about 1/2 mile away. We left early. The fire continued into the neighborhood after we left.
Disaster planning isn't suppose to be "we can take all the stuff out when the problem happens". It's suppose to be "Some spontaneous event happened, and we've lost all of our equipment." It should be dramatic enough to believe the building spontaneously collapsed, a meteor the size of a cit bus fell on it, or a tornado blew it away.
So, to address the summary, plan for the equipment to be a total loss, and that you'll have to restore from off-site backups.
What I'd do, if I happened to be there, would be to grab the servers, the C-level and Director level workstations (abandon the monitors, keyboards, and mice), and leave the rest. Hopefully everything is on the servers. It's not practical to consider moving 50 workstations, monitors, keyboards, and phones. I'd sure hope their insurance is up to date. We all know that top brass will pitch a fit if their computers are destroyed, which is the only reason for considering them.
There are probably at least a few departments who will need their files (accounting, HR, and sales). For most businesses, that'd fill up the 17' truck pretty quick by itself. That's a good time to talk to the boss about the paperless office concept, off-site backups, and a proper disaster plan.
Re:Uh... (Score:4, Insightful)
The Black Saturday fire front here in Oz was moving at 120kmh due to gale force winds coming off the desert, it did most of the damage in a single afternoon. Spot fires were being ignited 20km downwind from the main front which was creating its own local winds due to the massive column of hot smoke rising 15km into the atmosphere. The 3000degC heat from the fire front was leathal at a range of 200 meters, one (un)lucky survivor had his windsceen melt and drip onto his legs while he was taking cover under the dashboard wrapped in a heavy wool blanket. Another similar bushfire I wittnesed up close and personal was the Ash Wednesday fires, both fires continued to burn more or less harmlessly for weeks after the damage was done, at one stage the AW front streached unbroken from Canberra to Mallacoota (~500km)
Yes you may have a few hours warning if your lucky (ie: phone/power/repeaters are still working), but most of time the only warning is a lot of smoke and no bird noises, it makes the roads just as confusing as a smoke filled building, you have no idea where the fire front is or which way it's headed until a 100m high wall of fire runs over you at high speed. Admittedly the Black Saturday fire is an extreme example but a bushfire front moving at highway speeds is not uncommon in this country and I believe smaller fire fronts in California can also reach similar speeds when driven by strong desert winds or running up the side of the Hollywood hills. Footage I've seen of these looks very similar, I recall seeing a yellow timber mansion sitting on top of the ridge, it was being filmed at a distance by a helicopter crew, it's heavily smouldering walls literally exploded into flame before the front even got to them. Much like a piece of paper bursts into flame when held above a candle flame.
Of course as you imply the smart thing to do for any fire is to have a plan before the fire starts, If you invest some non-trivial thought, money, and effort into things like not putting your mansion on top of a natural blow tourch, water tanks, roof sprinklers, pumps, dug-outs, fire-proof windows/shutters, etc, you and your property can stay and fight the worst firestorm, it's a safe bet that at least you will come out the other side relatively unscathed. If you're like most people and your plan is to evacuate, then evacuate, don't fuck around loading a 17 foot trailer or looking for the cat. The servers can be replaced, the trailer is a fucking Albotross on a smoke filled road where you may have difficulty seeing past your hood ornament and may be directed to turn around by fire fighters, and the cat can cope with a bushfire much better than you can.
*You - Not 'you' the AC, rather 'you' the Darwin award candidate that posted the question. A final piece of advise from someone who has wittnessed several serious bushfires first hand, if you are already working with the local fire brigade then ask them for advise on an evacuation plan in the case of bushfire rather than slashdot. I'm certainly no expert but looking at the immediate topology around the center (in 2d via google maps) it looks like the property sits on a small hill that itself sits at the bottom of a large natural bowl in the mountains, it also has a creek or river on one edge. Maybe with their help it would be relatively simple project to set up and defend the property as an official refuge point for the area. Which, if I'm guessing right, would be the kind of thing the center would be eager to advertise to their customers and insurance broker.
Re: (Score:3)
Offsite back-up is right, like in the cloud. That way, once they get new equipment, they can always restore what's been lost.
If it was an emergency, such as a current fire, what you say is right. But if it is one of those wildfires which is spreading and which threatens to toast that place, and they do have time to salvage and retrieve their equipment, why not. Although I find it strange that they can't just get enough packing equipment from U-Haul, which in addition to trucks, does sell packing suppli
Re:Uh... (Score:5, Insightful)
Mod parent and grandparent down, please, and tell both posters what doofuses they are. In hurricane country, do you think people are stupid for taking time to board up windows before leaving too? No. Spending time saving property when a natural disaster or like phenomenon is known to be on the way but remains hours or possibly days away from impacting you is, in fact, a DESIRABLE thing to do. Less property is destroyed. Less time and effort is spent replacing the property. Less time is spent filing insurance paperwork.
Perhaps you're in California and have earthquakes on the brain. It's quite different. This is not "get out or we're all going to die" situation. This is a "be ready to leave town if we tell you to" situation. Sure, if the guys in charge of evacuation tell you "leave IMMEDIATELY omgfire" then you do, and save the people, and throw away the property. But it doesn't always come to that.
Re: (Score:3)
Your'e looking at the wrong scale. The company needs a complete contingency plan for all kinds of threats. The OP's question is just a small part of his problem.
I've seen buildings burn down. I've talked to the guy who went into a burning building to rescue the backup tapes. Do you want that to be in your job description?Planning for failure is not fun, but it's part of the job.
Re: (Score:3)
Mod parent and grandparent down, please, and tell both posters what doofuses they are. In hurricane country, do you think people are stupid for taking time to board up windows before leaving too?
If it means buying the boards and hammer at the time you hear about the hurricane? Hell, yes!
You are then one of those who jeopardize others by being the last ones out, causing endless non-moving lanes of traffic and broken down vehicles.
I hail from a port where every house had iron bolted storm shutters, and sturdy ropes to string between the houses. That's preparedness. Boarding up windows is panic due to lack of preparedness.
Re:Uh... (Score:4, Insightful)
Can we change
"Save your data and your people"
to
"Save your PEOPLE and your data"
??
Re:Uh... (Score:4, Informative)
Agree. You evacuate a network the same way you evacuate your building's other utilities (water, electrical, furniture) -- you don't. That's what insurance is for.
Your insurance company will pay to replace anything that's damaged by the fire. They probably won't replace anything that's damaged as you evacuate and re-occupy, or for the work needed to put it all back together. Yes, this is a "moral hazard" situation, but that's not your problem.
Re:Uh... (Score:4, Interesting)
Looking at the Map [goo.gl] there is nothing to worry about.
His building is far from any significant stand of trees. Two guys with chainsaws and another driving a 4 wheel drive truck can
drop every tree close to the building in 20 minutes, and tow them to an open field.
Use a backup generator to keep his well pumping (if no city water) and put a lawn sprinkers on the roof.
One wonders if this wasn't just out out there to drive traffic to his website.
Re: (Score:3)
He still makes a good point. I've read about fire prevention tactics. Turns out that we CAN build stuff that survives the worst wildfires possible, to the point that firefighters have sheltered in them in emergencies. Heck, in my time with the department they were getting tents that could protect a single fighter through a grass fire(nasty fast) that can be deployed in less than 30 seconds by said fighter.
The sprinklers aren't to stop a fire right alongside the building, the sprinklers are to keep blown
Re:Uh... (Score:5, Informative)
Your insurance company will pay to replace anything that's damaged by the fire.
Insurance does not cover any damage that should have been prevented or was caused as a result of culpable negligence, when the org intentionally passed up a reasonable opportunity to mitigate or prevent the damage.
If you had an opportunity to mitigate or prevent the fire damage because there was sufficient warning, and you intentionally avoided mitigating the damage, that a reasonable person would have taken actions to prevent, then your reckless inaction likely means that the insurance company is not obligated to pay for the fire damage that resulted from your inaction.
Emergency planning (Score:3)
What I'm seeing here is a disagreement on the time frames involved. I liked one of the posters who mentioned 'acute emergency'. A wildfire miles away, with the area being declared a 'pre-evacuation' zone indicates that there is not only no exceptional danger yet, but that there's an emergency alert system in place to tell you BEFORE it becomes acute.
In which case acting to protect property is still on the cards. Choices include 'shelter in place' and 'evacuate'. For a fire, the answer is generally evacu
People first, data second (Score:5, Insightful)
Save your data and your people
but not in that order.
Re: (Score:3)
Let each employee take care of his/her workstation and let the IT staff worry about the servers and data.
And do a quick prioritization - machines ripe for upgrade anyway can be left behind.
Re: (Score:3)
Triage and Labels (Score:5, Informative)
Log off Slashdot before reading this :-)
If you've got labeling stuff around, use it (not fancy label makers, just the basic "Hello My Name Is" and a Sharpie.)
Grab the servers, grab the workstation bodies, grab the phones and anything else that's easily portable, and any backup media you've got. Unfortunately, rack-mounted equipment is usually harder to grab, but that's probably your most expensive and critical stuff. And it'll be your critical path, so start unbolting it first. All of that will fit, put it in first, braced as well as you can.
Monitors and keyboards are nice, but they're just money, not data. Grab a few of them, but leave the rest for last. If you have packing material left, great, but if not you'll just have some breakage. If you've got any CRTs, leave them, they're heavy.
Re:Triage and Labels (Score:5, Insightful)
dont unbolt the parts it's time wasted and it will make transporting the lot easily while in rack with a buggy .. .if the cat 5's are landed at punchhed patch panels
just unwire the rack and take it out as a whole
you may be able to remove the cabling in large chunks without causing too much damage if any.
computers ? same as above. just take the stations and put them in large bins..If you got an apple producer
of similar large produce cases you may be able to fit all computers in one box which again is handy because
it keeps things together and the screens in the second box . Dont waste time on kb's mice etc unless you have a
lot of time on your hands..
get the heck out and keep people safe is first
hardware comes last.
it's useless to dead people
ric.
Re: (Score:3)
Hadn't even thought of moving racks - here in California, they're usually bolted to the floor and often also braced at the top, for earthquake reasons, as well as for "not falling over when you slide servers in and out" reasons. Forgot that not everybody does that (and also, only most of the equipment in my lab racks is actually screwed into the rack - some things are on shelves and will slide around, especially smaller-than-1U devices, power bricks, etc.
But no, moving whole racks is likely to be a really
Re:Uh... (Score:5, Insightful)
Wow, way to plan for disaster. You should already have had the systems transfered either to a clowd or to your remote site. If you do not you've failed.
So what this gentleman says is correct. If you've not labeled everything including cables and have detailed drawings of the installations wiring you've failed.
So what you do is get out your label maker and tools, shut it all down and label everything. Then pack it as best you can in the truck. You can expect 30-40 percent startup failure when you get them installed and attempt a startup.
You might just want to consider building your next IT center in a shipping container that can be detached and loaded on to a semi. Done properly your UPS and AC systems would keep them alive until you could get to an alternate location with power and network which you should already have contracted for in advance.
Re:Uh... (Score:5, Insightful)
You might just want to consider building your next IT center in a shipping container that can be detached and loaded on to a semi. Done properly your UPS and AC systems would keep them alive until you could get to an alternate location with power and network which you should already have contracted for in advance.
Nods, and seconded... motion to carry.
Re: (Score:3)
Quickly?
Yeah, just get them in the van.
As fast as possible. Too late to make fancy plans now.
At most you can sketch a floorplan and number them with a sharpie as you grab them.
Re:Uh... (Score:4, Insightful)
Welll... (Score:5, Insightful)
Re: (Score:3)
Take the hard disks and RUN!
Re:Welll... (Score:5, Funny)
And leave all of the Mountain Dew behind. When the fire hits, they will heat up, explode and smother the flames.
YES! Save only hard drives (Score:5, Insightful)
1. label all hard drives. Drive position, and server.
2. place hard drives in anti-static bags
3. pack drives in foam.
4. get drives far, far away.
Hard drives are both the most valuable, and the most fragile part. Do not load them in a stiff suspension vehicle like a truck, as this bounces the drives. Choose a soft-suspension normal car.
Next take servers and network gear. Desktops are a maybe, as are phones. Ignore printers.
Tape a piece of cardboard over the face of an LCD monitor to protect it from casual bumps.
Above all, no data is worth a human life. No heroics. You're not paid for heroics.
Re:YES! Save only hard drives (Score:4, Interesting)
Hard drives are both the most valuable, and the most fragile part. Do not load them in a stiff suspension vehicle like a truck, as this bounces the drives. Choose a soft-suspension normal car.
I know someone who uses a Citroen Xantia estate with the hydraulic suspension modified to be slightly softer than normal for moving delicate optical instruments. It just comes down to a little adjustment of sphere pressure and damper ports.
Even unmodified, if the suspension is in good condition you can't even feel speed humps at 60mph, just hear the "ba-dunk" as you go over them.
Re: (Score:2)
label all hard drives. Drive position, and server.
In a "server room" situation, wouldn't the HDDs be in rack-mounted trays? If so, just take the trays. In fact, *only* take the server and HDD trays.
Everything else can be replaced from fire insurance. (You are backing up all the workstation data to servers, right?
Re:YES! Save only hard drives (Score:5, Funny)
Ignore printers.
Most printers ought to die in a fire anyways. Now he might actually get to see it happen!
Re:YES! Save only hard drives (Score:5, Funny)
http://en.wikipedia.org/wiki/Lp0_on_fire [wikipedia.org]
Re: (Score:2)
Prioritize (Score:5, Insightful)
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
Re:Prioritize (Score:4, Insightful)
I have to agree prioritizing is key. Fires are live creatures and can go from bad to disastrous in a flash. Get the data first which includes databases, file servers etc.
When it comes to packing if you have limited packing supplies focus on the most critical and hard to replace stuff first. If you don't have enough stuff to package every thing then at least make sure that nothing can move around or fall over in the truck. All most every thing is built tough enough to handle a trip down even a mountain road as long as you drive slow and stuff isn't falling over and rubbing against each other. Every thing can be a packing supply. Coats, boxes. blankets, carpet, string, rope, cables etc.
Good luck and if you feel up to it give us an update when you are done.
Prioritize efficiently. (Score:2)
Make sure you have off site backups of everything needed to reconstruct your network.
After that it really doesn't matter. Either you can move everything out in time or you cannot. If you cannot then you move the people and forget the gear.
Just like in the fire drills for almost every other company out there.
Re: (Score:2)
Make sure you have off site backups of everything needed to reconstruct your network.
At this point that would a "shoulda done" thing. I'm guessing since he's asking, they don't have a clue. Hopefully you practiced proper source control/workstation backups. Grab the servers and place them in a car(s). After that, the most important workstations, any truly expensive pieces of networking gear, and then whatever else you can. Realize anything in the U-Haul may not survive the trip, even with packing material.
Re:Prioritize efficiently. (Score:5, Insightful)
Re:Prioritize (Score:5, Insightful)
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
Quick-disconnect hard drives. Everything else can be replaced by insurance, but your data can't. With what you've got listed above, I could hike out with your company in my backpack. The other thing is, consider the health and safety in your disaster recovery plan -- you should not expect, nor ask, your employees to stay until the last possible moment packing in equipment. Equipment can be replaced... lives cannot. Nobody should ever risk their life for an inanimate object in a business environment.
The other thing is, you should have a disaster recovery plan that includes regular backups to an offsite facility. Any disaster plan should be able to cope with "and then a giant foot appeared above the building and squished it flat." Yours should be no different. It might not be a wild fire that threatens your servers... it could be a UPS that shorts out, or a tornado, flood, a failed fire suppression unit, or simple human incompetence (Yes, I've seen stupidity kill buildings).
Any plan that relies on people staying in danger to save your business unethical, immoral, and probably illegal. So save what you can reasonably and without risk take, in descending order of importance... but recognize that there may be situations in which the only solution is to exit the building at a dead run and not look back.
Re:Prioritize (Score:5, Informative)
They're in a pre-evacuation phase. Smoke on the horizon and all that. Not to worry, the local sheriff department will kick people out long before there is a problem. They've done this before. Like every year in the past decade.
Re:Prioritize (Score:5, Insightful)
They've done this before. Like every year in the past decade.
So the answer to the Ask Slashdot question should be: "Just refer to the emergency evacuation plan you drafted years ago, and have regularly updated in the meantime."
And if that isn't the answer, why the hell not?
Re:Prioritize (Score:5, Insightful)
Right idea, missing a detail. Get your data and hard to replace equipment (e.g. custom orders, long lead time, no longer available) first. That may be servers, or just the HD's from them. After that, everything that is replaceable can be picked based upon it's value, size, ease of removal, and available space. If you have to take workstation HDs, try to get one of each model of workstation so you have at least one machine that you know will work with that HD. It's not critical, but it can save you some effort if the facility does burn. Most networking gear, phones, workstations, etc. are easily replaceable, don't mess with them until the more important stuff is out.
And most importantly, DO NOT WAIT until you receive the evac order, start packing at least 24 hours before an evac is likely. I don't care what management says about taking down the network early, your data and your lives are far more valuable than an extra day working.
Re:Prioritize (Score:4, Insightful)
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
I agree. But I would prioritize slightly differently:
1. Make sure all non-critical staff are gone, and are well informed of what's going on -- where they can contact people for further information, etc. (People)
1a. Whlie you're talking to them, ask the managers of departments -- off the record, of course -- if there are any department specific, hidden fileservers that need evac. I haven't worked in an office yet that didn't have at least one.
2. Make sure all backups are offsite, preferably in a dry, fireproof safe someplace. Ideally this step happened years back, and you can roll your eyes at this one, but lets be honest -- it didn't and you can't. (Data)
3. Disconnect servers from their racks. Any data storage stuff in there takes priority. (More Data)
4. Rack mounted servers go next (Servers)
5. The rest of the server room as time allows (Networking gear)
Anything after this is probably stuff you can skip, assuming you have good fire insurance. If you don't, welp. Honestly, start thinking like a thief, prioritize things that are expensive:
Harddrives are good to try, but it's easier to just pull the towers. Aim for any high end workstations -- the secretary's machine probably shouldn't go (but be aware that they may not have followed your server file storage and there may be data on that workstation not on the server), but the guys back in marketing? Maybe that top of the line workstation with the 30" monitor may need a second look. As mentioned above, many companies will have unofficial servers hidden around or local backups of department specific stuff, make sure you ask around if you have time to see if there's a file cabinet that needs placed on a dolly.
In an absolute pinch, just use wire cutters to disconnect workstations and get them on a cart -- DVI and USB cables are cheap. Monitors are next up on the price list. Printers right afterwards.
If you do not anticipate fire actually taking out the buildling, it may be prudent to grab trash bags and cover monitors and towers with plastic instead. This will help keep any smoke or sprinkler systems from pouring on them and damaging things.
If you have a basement, or a fire proof safe, tossing stuff in it may save it if you are absolutely out of time.
Re: (Score:3)
Pack what's critical first. Servers.
No, servers aren't critical unless they're irreplacable legacy systems, in which case you probably already have a spare standby elsewhere.
Humans are.what's critical Get them out as quickly as possible.
Then data - remotely start another adhoc offline backup and leave it running while you get out.
Remember that if the National Guard tells you you have half an hour to evacuate the premises, it doesn't mean you should aim for half an hour, it means that you should evacuate as soon as possible but under no circu
The site (Score:5, Funny)
Um... (Score:5, Funny)
Relocate (Score:3)
Offsite backups (Score:2, Insightful)
Damn if I'd go in to work to remove hardware when a fire is threatening.
I'm not paid enough to risk my life. Period.
Pictures (Score:5, Interesting)
Take lots of pictures before you unplug your cables. It will save you time when you have to reconnect everything.
Re:Like pictures are going to help on reconnects (Score:5, Insightful)
http://eddie.niese.net/images/Wiring_BAD.jpg [niese.net]
You don't. (Score:5, Insightful)
The "best" way to evacuate a data center is to already have off-site back-up for your data in place, drop a fresh copy to portable media, and walk out. The hardware should be insured. The life of your and your people (at least some of whom should probably be helping their families evacuate) are far more valuable than a few months of making your insurer pay for rented hardware until your new machines show up.
Re:You don't. (Score:5, Interesting)
The "best" way to evacuate a data center is to already have off-site back-up for your data in place, drop a fresh copy to portable media, and walk out. The hardware should be insured. The life of your and your people (at least some of whom should probably be helping their families evacuate) are far more valuable than a few months of making your insurer pay for rented hardware until your new machines show up.
Well, it's obvious the poster here was handed the job of preparing a disaster recovery plan and has no professional experience doing so... probably was given the assignment by his manager who had no idea the complexities of the task. If the OP is in that position, then it's also likely they won't see any benefit to offsite backup, or they'll blunder by putting the offsite backups in the boss' house which is three miles downwind... assuming he can even convince them to budget for it.
In that case, I'd say buy some quick-disconnect drive enclosures (the kind where you lift a lever and a harddrive is now dangling in your hand), write a formal letter of protest outlining exactly why you're not responsible for the company being wiped out, what mitigation steps you'd recommend with a proper budget, and keep a copy in a safety deposit box or some 'cloud' service far, far away from you... because yeah. -_-
Story time! I worked for a Fortune 500 company that connected consumer-grade 300watt rated UPS to racks of equipment... they were unaware of the risk of fire until I explained to them that with 2,000+ store locations and about 50 distribution centers, and 3 corporate headquarters, while the odds of any one of them failing catastrophically due to current overload was low, each one of those buildings experiences a 'power loss event' an average of a dozen times a year... so it became very likely that they would fail and cause a fire, which wouldn't be covered by insurance. Management tried to ignore it, but somehow (wink, wink) legal found out about it, and forced the Board to fix the problem post-haste to avert having to pay 50 million plus to rebuild the burned out husk of a store after the fire chief finds the flash point was a piece of equipment that was massively under-rated for the job.
Disaster planning requires a good understanding of probabilities and statistics. That understanding is surprisingly rare in the business world, despite what most people think.
Re:You don't. (Score:5, Informative)
This. The parent is already +5 Insightful, but really needs to be +500.
You don't evacuate a datacenter, you abandon it. Any other plan is a dozen different kinds of stupid.
At best you trigger a self-destruct (software or better yet hardware) to whip all data so scavengers don't get to it while you're fleeing.
Hardware can be replaced easily (insure it, duh). Lives and Data can not. So already have the data backed up offsite and let the lives flee as they can at the first sign of danger w/o being hindered by insanely stupid commandments like "save the copier!!!".
Scavengers? (Score:3)
Re: (Score:2)
But if it's already late for that, pull the drives for safe packing. The rest can take it's chances.
Triage (Score:2)
Emergency packing (Score:5, Insightful)
First, triage the equipment.
You likely do not have time to pull disks from systems, so pack computers and
external drives first. Get blankets to protect things. Blankets start at the bottom
to act like a shock absorber.
Things like networking gear and wireless stuff is irrelevant compared to the
computers, and probably lighter. If you CAN, sure, save all that stuff too.
But the data comes first. Don't forget backups.
If there are computers with really really important or sensitive stuff, put
those in someones car in the backseat, again with blankets. If I seem
blanket obsessed, it's because I've found them to be available quickly
either from individuals or stores. Yes, bubble wrap or sorbathane would
be better but you aren't likely to have that stuff lying around.
Re: (Score:2)
Wrong Time To Ask This Question! (Score:4, Informative)
Not trying to be mean, but you should have already had a plan in place...this late in the game without a plan means you just have to go with asses and elbows (just get what you can while you can and forget trying to install a plan to do it!)
The good news is you can become an example for other IT people! Everyone should look at their disaster plans and make sure you have accounted/planned for all emergencies that may happen in your area.
Re: (Score:3)
Having disaster plans costs money, most companies aren't willing to spend. For a company to have proper procedures in place it needs to have smart people at the top or government required facilities, both very rare...
Re: (Score:2)
Triage? (Score:2)
Start with what you can least afford to lose. (Score:4, Interesting)
Whatever stores data first -- if it's a SAN, then your RAID chassis and metadata controllers, and if you have time, the SAN fabric switches and cabling, but you can replace the latter if you have to, and if it's ordinary SAS, the servers if they're all internal storage, or the RAID chassis or whatever's external. Definitely grab any non-offsite backup media with that. Rest of it in descending order of priority after you grab the most valuable stuff, mostly to avoid having to replace it.
Best strategy overall is to think "what if we had to abandon this evacuation mid-process and run?" Try to have what you most want already in the truck at any given moment, and concentrate on data before hardware -- the data is far more valuable in most cases.
If you haven't done an offsite backup, for god/dess' sake do one *now* and get the backup media to a safe location .. :/
Cushion (Score:3)
Many servers and disk arrays specify that drives have to be shipped separately simply because their mass is so high that the rack doesn't pass shock and vibe tests. So either pull out the drives (mark which slot they go into) and pack them separately in bubble wrap or, if that's not an option, put as much cushioning as you can around the servers and strap them down so they don't bounce. In any case, be prepared for some disk drive damage or degradation.
have everything "ready to go" (Score:5, Informative)
thinking this through, your priority is to make it as quick as possible to get essential gear out the door. so, consider:
* having the servers and desktops *already* in easy-to-carry crates, with large handles on the outside and packing materials surrounding the machines.
* have the machines stacked off the ground so that people don't have to waste time bending down and possibly injuring themselves by jolting weight that's too much for them
* have all essential equipment nearest to the doors plural, prioritised by criticality
* yes doors plural: add an extra door next to the existing one (or replace the one door with easy-to-open double-doors with those pushable handles) so that at least two people side-by-side can get through at once, carrying the crates, and can just "barge through them" rather than having to twist the handles.
* make sure that the crates are stackable and sturdy but also light enough to carry!
* even consider having the machines already loaded onto 4-wheeled trollies and left on them, permanently.
* if time is _seriously_ critical, consider putting guillotines next to all cables (and test them) so that people don't have to waste time unplugging cables: just cut them and go - but only consider this if the guillotines are sharp enough and easy enough to operate, and only if it's seriously seriously critical to save seconds. don't put power cables through the guillotine though!
* consider getting convenient light-weight but sturdy cabinets made for all LCD monitors, with double doors that fold back 180 degrees out of sight, and a top (with a handle) that locks automatically when it's flipped over. have the LCD monitors mounted onto the cabinets with rubber bushes so that they don't need to be placed or positioned into the cabinets - just pull out the cables, shut the doors, slam the top over and pick it up by the handle: done.
* consider getting 12v powered LCD monitors instead of 240v/120v AC mains, so that the power cables can be guillotined rather than pulled.
* instead of guillotining, consider breaking all the tabs on the network and telephone cables (the ones that "click and lock") and affixing them *loosely* with gaffa tape to all devices (network hubs, machines etc.) - this way it will be possible to just pull (hard) and out pop the cables. or, if someone forgets, and gets to the end of the wire, they won't trip or be yanked backwards: the cable will just come out, clean.
* get 4-port hubs instead of 8, 16 or 24-port. 4 gaffa-taped cables are easier to pull out than 8, 16 or 24, and if one of the 4-port hubs is lost to a fire, so what, big deal. a 24-port hub however starts to get expensive.
* stop people from putting the bloody screws in the bloody cables - you know the ones: parallel ports, VGA cables, serial cables etc. the ones that are always bloody irritating when it comes to fixing or moving a machine and you find that the bloody VGA cable needs a bloody screwdriver to remove the damn thing. take the screws *OUT* of the cables; that way people can't go "oh look: screws - let's tighten them".
so - yeah. make it easy to just shift everything. have practice drills. set a deadline (say 1 minute) and see how much kit people can get out in that time, without damaging it.
oh - and you know how i suggested making it easy to shift everything? uh... make sure the insurance is up to date, and get good security. no point making it easy for *other people* to shift all that expensive gear, eh? oh. and sort out some off-site backups, eh? :) i use rsync; my friend uses backuppc (because he has a lot of machines). /peace
Re: (Score:2)
* add an extra door next to the existing one
Would that be done by driving a truck through the wall?
It doesn't sound like there's time to do the paperwork and get full planning approval to put some extra doors in...
Leave the equipment, take the cannoli. (Score:4, Informative)
Walk out with your backups, and save the people first.
I don't see (Score:3)
Consider this a learning experience (Score:3)
Had you adopted "The Cloud" sooner, this would not have been a problem!
Re: (Score:3)
been there done that (Score:5, Informative)
Two years ago we go a call telling us the levies might not hold and if they burst (1 block away) we'd have 8 ft of water. We didn't really have a battle plan and we had a lot less to deal with than it sounds like you did, but we learned some lessons.
1) praise the lord we had good network documentation. Now is not the time to be writing down how the firewall and public and private LANs are plugged in together. Shut stuff down, and start placing network hardware in big plastic tubs. Have tubs handy for this, they nest nicely when not in use. Toss cables in a different tub. just wind them up best you can into loops and toss them in. there's probably not time for neatness, you can deal with that later. TAKE THE DOCUMENTATION WITH YOU. You'll feel mighty silly if that's left pinned on the wall. or I assume you have an electronic copy you can print when you get offsite. Make sure any servers with complex cable attachments (like to phone banks or security systems) have labels on the connectors.
1b) got your phone system documented too? this is a whole 'nother can of worms that often is forgotten about. Does anyone have a diagram of where all those punched down wires go on each block? If you have phone switching hardware to pack, make sure the cables are labeled, they will all probably look the same with the giant connectors that attach to the blocks. "We'll just call Al, he does our phone stuff." Oh, you don't think Al is going to be BUSY helping everyone else that is returning? Nothing's as fun as a 2-3 day wait to get your phones back up and running huh?
2) Label ac adapters. You need to know which unit wants 12vdc and which has 24vac, you don't want to fry stuff when you are trying to reassemble. every pack should have the model of the unit it goes to written on it. Gear WILL get separated from its pack during the evac.
3) label staff's hardware. It's very annoying trying to figure out whose beige box is whose later. and they will probably fight over monitors and keyboards later. save yourself the headache. If you are already under the gun, run to the store and get a dozen rolls of masking tape and sharpies and have the staff label their equipment while you're packing things up, full initials or names, I bet you have duplicate first names you don't want to deal with later. Make sure you label the phones.
4) have a plan for things you can't easily move. the corp office was also forecast to get 8ft of water and they were on the WRONG side of the dike so it was more of a "when" than "if". they had a very expensive multifunction printer that the service people told them they could have a tech out to take it apart (so it fit out the door) in three days, which obviously was silly. They rushed in a bunch of cinder blocks and lifted it up and set it on them 8.5' up. (I have no idea how they lifted it) In retrospect, the building got 14" of water and totaled it, they SHOULD have killed power to the building and took a saws all to a wall. OR at least watertight wrapped it before lifting. I've seen this done with entire cars when faced with an incoming flood or hurricane. Even if it doesn't keep out the water 100%, at least it will keep out the mud, which you may be very grateful later. Got a plan for your big server room ups's? those can be quite large and heavy, and are often hardwired into the AC, are you able and qualified to unhook it? Maybe you should call in an electrician now and change that armored cable to a dryer type plug? Have a place you can move big stuff that can't be evac'd to where it will be at least more likely to survive. Think of flood, fire, and tornado/hurricane, there's probably not one single place that will work best in all three cases. Smoke damage can be very destructive, simply having something wrapped in mover's visqueen may prevent unnecessary loss that the fire missed but the smoke got. Do you have a plan for that rack that's bolted down or won't even fit through the door?
5) Document what's been left behind. A simple way to
Re: (Score:2)
They lifted it eight and a half feet, but fourteen inches of water reached it?
Only one possible answer. (Score:2, Funny)
Bring in an expert [youtube.com].
Computing on the run. (Score:2)
mattresses--servers--desktops--phones--screens (Score:2)
in order of importance, fragility, price and density.
put some spare mattresses on the floor/sides of the uhaul and put your servers down there. next to each other. (you do have mattresses, right? you're a retreat center and a big fire is coming...)
next desktop boxes, lined up next to each other.
on top networking and ip phones, combined into a few bags/pillowcases etc. these, particularly the phones are light and wont damage each other.
next screens, wrapped in blankets and stabilized. you'll find the scre
Professional Movers (Score:2)
Better to hire professional moves who can come in, grab the critical stuff, and pack the truck so that nothing gets damaged. Probably stuff like equipment racks can be dollied out in one piece and tied down in the truck - forget pulling individual drives.
Colo and insurnace (Score:2)
Too late for that now I suppose.
In lieu of that grab the drives and run.
Re: (Score:3)
Too late for that now I suppose.
In lieu of that grab the drives and run.
Second this. If you're thinking about this now then it's too late. Forget all of these idiotic elaborate answers on how to spend time in the path of a wildfire "evacuate" the network as they're likely to get you killed. Wildfires are extremely dangerous and fickle things. You may think you have 30 minutes, but then the wind shifts slightly and you're dead. Grab some drives and run is pretty much what you're left with.
Aside from that, your insurance may cover equipment loss by fire but it may not cover damag
Move in a Flash (Score:2, Insightful)
As everyone has stated so far, WTF are you doing on Slashdot, when your in an emergency situation. Now, onto my part, WTF are you doing as a IT Manager, and you don't have a critical battle plan, How the hell did you get this job in the first place. Your previous background was not in IT somehow. /done.
First thing first, Grab the essentials. Since you stated 50+ workstations 6 Servers and random IT 'junk' then your most important priority is obviously the data. Whatever your co
Modularize (Score:2)
Portability (Score:3)
Laptops/notebooks, tablets where possible. Anything that can be used in a zipper case is a plus.
Wireless of course. Several sets of access points, two in the trailer. The production ones can be left behind to burn.
Many of today's notebooks can do a good job as a server, remember to use power settings that make sense for server duty.
Backups of course, but probably external drives.
If you have the flexibility to choose your server OS, one that offers a resilient filesystem is good, since you may have to power down in a vehicle of some sort. Pulling the external drive off when running will exercise the resilience.
Bugging out would consist mostly of closing lids, zipping up cases (maybe) and running. Servers go the same way. IF you can grab the APs, fine. For 50 users, this will not take a half hour. Crates to take the zip cases should let you essentially drop stations in there. You can build crates that cushion your servers better. External drives get better cushioning also, but using notebook drives enhances their durability, some. Everything else can survive.
Keep a set of UPSes in the trailer, or at least by the back door, charging. These will get handy when you arrive at the new location, and if you save the old ones, you may use two sets to give you instant power while you get everything running, and find the outlets for permanent power. A generator would be handy, and it need not be big. Propane rigs are easier to handle than gasoline. If your evac point is within 2 hours' drive, you may even be able to safe the servers, park the drives, and take them on the trip running. You ARE writing scripts to do emergency shutdowns, safe modes, parks, and closing critical apps/saving data, riiiight?
I'm assuming you may not always have 24 hours' notice. If you will have a guaranteed 2 hour notice, then use short racks that can be wheeled around, and you can have fairly conventional servers and wired network, just plan on abandoning the cabling, which is entirely expendable IMHO. Leaving the servers and switches cabled together is helpful, and sme simplified interconnect to mutiple cabinets will help. Plenty of cables in the trailer, and a spool of cable with a bag of plugs and at least 2 crimpers also. And a simple tester. Trust me on this, no point in guessing if you made it right. Making those 200' cables to solve a problem would be handy.
Lots of diagrams laminated to the cabinets is handy, even a grease penciled fill in the blanks chart to show what was built is a blessing when you reconnect.
Somehow, I suspect the military has some advice for you on this. Someone in Interior or the Forest Service must have a contact.
I would love to be in that business. Nothing like having to make DR plans that have to accomodate the loss of the facility to sharpen your focus and get the juices flowing. The last project like that I was in, a financial institution needed a similar plan, and we even has a BOM at a distributor ready to be ordered and shipped on notice, updated quarterly. Almost got to do it for real, but they fixed the gas leak without blowing up the building. Darn. :)
consider shuting everything down right away as wel (Score:2)
unless you have offsite replication you may be better off telling people to stop working than have them work till the last momrent and then shutdown.
If you do nightly backups and some people do crital stuff this morning you can loose that work or restore to an inconsstant state. for some information it may be better to be down then inconsustant.
you dont want customers to have reciepts for transactions you do not have backups on your end.
Oooh! (Score:3)
Disaster Recovery (Score:2)
Packing rules should be made as part of your disaster recovery plan. But as a rule, I save the packing for large server and switching equipment. That way there is no issues with packing.
Workstations I don't save the packing after 30 days. In that case I would plastic wrap them and ask employees to assist if due to circumstances all available shipping options are exhausted.
Good luck.
Hack
Mobile setup (Score:2)
I know this won't help the OP now, but If you live in a disaster-prone area and you could ever be hours away from a sudden evac, consider setting up your hardware as if you were doing a mobile installation. 19" rack cabinets on casters, which can be quickly pushed to a truck. Note: just because you have wheels on your rack doesn't mean they're meant to be moved when loaded down with equipment. But the appropriate equipment. If you want to see something like this in action, go to a major televised sporti
Quick and dirty (Score:2)
I would leave all the cable behind really, no time to pack those up, just gets annoying. It might actually be easier to go through them with some hedge cutters (after you turn off the power off course).
If you have small racks to move, just make sure they're free to move and put them on a cart with casters, just be careful they don't tip over but a decent forklift will help with that. That only works for racks that are not fully used (like half racks or smaller). 42U racks that are filled can't be moved, get
two prongs to the fork here (Score:3)
(A) have employees take their desk stuff... the laptops/desktops. four wraps of bubble wrap, and go. screw the rest. there are usually a bunch of apps and work files on individual PCs that are irreplaceable. some are even business-critical. but it's THEIR machine, they're up now because it's set up and ready to go. add DSL/cable/whatever and they can work from home.
(B) send somebody west with backup set A. send somebody east with backup set B. then take out the servers and go. if you have very freaky setups in firewall appliances and routers, bring them, too. I like the idea of cheap-ass mattresses. fill the floor, and set the racks flat on their backs on the mattresses. yeah, it's overkill, but you have your inter-rack wiring, etc there.
(C) insure you have critical business papers... server and software licenses, articles of incorporation, insurance policies... with you.
screw the rest. employees can go anyplace for a monitor and keyboard/mouse kit. you can get a X-pack of twice-recycled cell phones at any of the corporate stores within 12-24 hours to operate on temporarily. if you can't get into building space at once and need to operate the business, haul the server room stuff to an ISP with colocation facilities and have your MX record transferred to their pipe.
Quick! (Score:3)
To The Cloud!
Re: (Score:2, Funny)
Doesn't cover pre-existing conditions (fire).
Re: (Score:2)
If the company goes under because they don't have backups and all the data is gone then insurance doesn't help.
If they've got a few hours notice then loading 50 servers into a van isn't a stupid idea.
Re:Simply Throw It In (Score:4, Funny)
- 1 UPS
- 1 computer (desktop/laptop/whatever)
- 1 printer
- 1 reem of paper
Now you have all you need to print resume's while driving the hell away from that building as fast as you can!
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Fully agree with this AC. If this is a geek/nerd posting this question you should be ashamed of yourself. I was expecting some sort of photographic evidence with layout and resulting HOWTO. HOWTO: Evacuate your data-centre quickly.
1. Highlight highest value equipment first
2. Arrange the physical design of the data-centre to quickly remove priority rack-mounted equipment first.
3. Prioritize: Stuff not currently backed up; expensive equipment.
4. Flood the place with flame retardant before leaving.
5. etc