Outfitting a Brand New Datacenter? 110
An anonymous reader writes "We completed our new 4,000 sq. ft. data center (Tier II/III, according to The Uptime Institute) and just recently moved our core systems from our old data center to the new. We've been up and running for several months now and I'm preparing to close out the project. The last piece is to purchase some accessories and tools for the new location. The short list so far consists of a Server Lift, a few extra floor tile pullers, flashlights and a crash cart. We'll also add to the tools in the toolbox located in one of the auxiliary rooms — these things seem to have legs! What are we missing? Where can we find crash carts set up more for a data center environment (beyond the utility cart with and LCD, keyboard, and mouse strapped to it)?"
hmmm (Score:1, Funny)
Breathalyzer (Score:1)
Safety equipment (Score:5, Insightful)
O2 masks for when the Halon drops
arrows on the floor directing people to the nearest exit
a 'Battleship' style row/column marker for every row/column of racks
near-Draconian access control policies
Re: (Score:2, Informative)
Re: (Score:2, Informative)
Re: (Score:3, Insightful)
Tell me something - if you're going to ask for casual advice on outfitting a data center, WHERE BETTER to ask than Slashdot?
I've built 2 from the ground up now. I don't mind sharing little nuggets.
When I have a question about how to put the finishing touches on a house, I don't see the harm in someone going to a home builders forum and asking for casual advice. Of course you want professionals doing the bulk of the work, but it's the little things that always get you, and we can c
Re:100% IDIOCY. Hire a ***ing consultant! (Score:3, Interesting)
First off, it's not like the guy is asking for advise on critical systems like cooling or something. It's more like a last minute gizmo checkup(or a way to rid themselves of budget leftovers?) Some people can come up with things the "asker" or the other readers hadn't thought of - for example, it would never occur to me about tarps in a DC(credit: a few posts below) because mine is sandwiched betwe
Re: (Score:2)
Safety equipment++ (Score:4, Informative)
Drop it across the terminals of one of your backup batteries -- when it's disconnected from the grid. When the wrench cools off, store it in a safe place. Makes a great scapegoat when things go wrong. Could save your career...
Re: (Score:2)
Make sure your fingerprints aren't on them?
Re:Safety equipment (Score:5, Insightful)
Lots of cheap plastic tarps that are stored out of the way but that you can deploy quickly in the event of a water event.
I know, you think you'll never use them, but if you do (storm leaks, broken pipe above, etc...), they'll be the most valuable tools you could have spent $100 acquiring a whole bunch of.
You just haven't lived if you haven't empirically tested (even accidently) how long it takes for power circuits under six inches of water to blow, or how those drop ceiling flourescent tube lights look when they're full of water and still going, or how long servers and switches stay up with water pouring down the racks into them....
Re: Tarps (Score:5, Insightful)
tm
Re: (Score:3, Funny)
Re: (Score:2)
We also had storm water funnelled into the raised floor are where the power was all running. The outside flooded and apparently where some of the data conduits went out of the building, water could come in and then pour down onto th
portable aircon and door wedges (Score:3, Informative)
Re: (Score:1)
In a data center of this size (4000 sq. ft), with any significant density, a portable chiller and propping the doors really won't help.
I recently toured a Department of Energy research lab with a couple smallish but high-density data centers. (10 kw per rack; computing cluster with racks full of 1U servers).
They had designed
Re: (Score:2)
These are definitely my people in here....
And a shop vac (Score:1)
Tarps - Yep (Score:2)
Cover all the racks and equipment before you begin, or you'll be sorry. Fiberglass and/or gypsum dust circulating forever, grime and dust and rat droppings hurdling toward every fan,
Re: (Score:2)
Re: (Score:2)
Both chemicals are as designed to displace oxygen, and therefore both chemicals can cause dizziness and disorientation in small quantities and death in large quantities, but they are both non-toxic and easily dissipated. It's just that Halon-1301 has
Re: (Score:2)
Re: (Score:1)
and a policy making sure no one uses the floor tile pullers to rearrange those tiles with the arrows on them
Re:Safety equipment (Score:4, Insightful)
1. A list of what UPS' take what batteries, and contact with a vendor that can get them to you ground in one day without having to pay overnight shipping. This is priceless. Trust me. If you have a full-center battery backup this actually makes it even more important, not less.
2. A *REPUTABLE* generator service person, and a service agreement. I never realized how hard it was to find someone that was reliable, and that when they say "24-hour service" it means that someone actually answers the phone, and responds withing an hour, or heck, even a day.
3. Lots and lots of screws, cage nuts, and cage nut removal tools. All of the prior have a habit of up and walking off. Have plenty on hand.
4. Hideaway bed. I'm not joking. Have a place where a tech can crash after a 2 am service call. Better than driving exhausted.
5. AdderLinkIP, phone line installed to core rack, and Modem. Hook the AdderLinkIP to your KVMs, and code in all hosts. Now you can get console remotely on any box, PLUS if your internet connection should go down, someone can dial into the AdderLink, and be able to troubleshoot the net connection from the inside without having to drive. Priceless.
6. If there is not a break area nearby with a fridge, get a mini fridge. Stock the fridge with caffeine. Make this a habit that the company provides. Caffeine without having to leave to get it is priceless during a crash too.
7. Anything else you can think of to make your network engineers LESS repulsed at the idea of having to be there. After a while being "the guy(s)" to go to when things go down has it's novelty wear off. If it's within an office, an arcade cabinet, darts, an XBox, ANYTHING....ask them what they enjoy doing in their off time. Heck, alcohol in small quantity is even done in our office (we have a micro-brewery across the street - I have 2 growlers in the office for those long weekend projects).
By saying you're Tier II/III, it leads me to believe you don't have to present as much of an image (hyper-clean room environment with lots and lots of racks, and nothing else), so do what makes you feel comfortable there. You'll be spending more time than you'd like there soon.
Re: (Score:2, Informative)
We've got redundant feeds from the power company, redundant diesel generators, redundant UPS systems, N+1 HVAC system, etc.
Image is important. We frequently have audits and they typically want to view the data center. Certainly a clean room type environment, with scheduled above/under floor professional cleanings and no clutter. No games, tool boxes, shipping containers or the like. Servers a
Re: (Score:1)
Re: (Score:1)
Re: (Score:2)
Please email (or reply) at bandman -at- gmail.com
Thanks!
Re: (Score:2)
lights that don't require wall-power; led stick-ons in critical places can be good
ear protection (that room is louder than you think)
extra tape and permanent markers in multiple colors for labeling.
high-intensity snake-type lights
mirror and magnifier for reading small labels on machines
clearly labeled signs "to Power, Cooling, Bob's Office", so no mystery cables
Lots of spare parts. Preferably 1-2 servers, switches, etc which serve only to be hot-swapped or cannibalized i
Re: (Score:1)
What are we missing? (Score:3, Insightful)
You've migrated several months ago - if you don't know what you are missing yet you either haven't been paying attention to what you need locally or haven't been paying attention to the recent news. Any small items you've probably already needed and know about.
Large things - like accomodating power outages (see 365 Main St) need to be prepared for. I'd guess after a successful migration you've likely covered most things.
Re: (Score:3, Insightful)
- one of those headlamp lights for hands-off work on servers (put this in the tool box)
- a way to track who has the tools in the toolbox (check it at start and end of each shift and record such)
- at least 2 cordless headset phones (ever try to move around a server room tied to a cord)
- a supply of batteries for everything that needs them
- a couple 7-day temperature gage chart recorders at various locations in the center + supply
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Amen, preach it brother. Can I get a hallelujah?
(and for some reason that quote is tripping the lameness filter. good grief taco....)
Re: (Score:1)
Re: (Score:1)
Gear you should have on site (Score:5, Informative)
Gear you should have off site (Score:5, Funny)
For those times when the internal security system is working, but not according to spec...
Re: (Score:3, Funny)
You missed out stuff like the shotgun though
Re: (Score:2)
Re: (Score:1)
Re: (Score:3, Funny)
You should see the look of horror on the users face when he did this. Management used to ask me if it was proper and I would tell them he was the senior tech so it must be. But it was hilarious to me.
Unfor
RE: THE Hammer. (Score:2)
After I got them out, the frame was rather disfigured, but salvageable, but I needed a hammer to fix it. Off to Home Depot I go...
What do you know...they have all kinds of small sledge hammers, but one stood out to me. It had a label on it:
"Engineer's Hammer"
Oh. Hell. Yes.
(and
Re: (Score:2)
When I tell a tech, "Hey, pass me Mjolnir!"
Nice crash cart (Score:4, Informative)
At my current and my past company, they work real well. I looked high and low for a good crash cart and nothing seemed to come close to these. Maybe I was just searching the wrong terms(and apparently my vendors were too). They are a bit pricey though, ~$1500 or so to start. I have a Styleview LCD cart at my current job, and had a LCD cart and a laptop cart at my last place (servers were co-located in a ~900 sq foot cage, 8 feet between rows, so plenty of space for the carts).
I also bought a KVM over IP/CAT5 solution from raritan(http://www.raritan.com/ [raritan.com]), which worked out real well for those situations where a serial console wasn't enough(unless you have fancy out of band management, some do, some don't). I setup tables in the front of the cage, hooked up a couple of the raritan hardware clients. Typically ran one CAT5 cable w/KVM hookup to each rack, so it could be plugged into any system fairly easily. Range of 1000 feet. This was pretty pricey too, with the adapters and all it was about $25k. Though in the grand scheme of things it was cheap at the time. I had cyclades terminal servers in every rack, with serial consoles on all the servers and network gear.
Also I hooked up a temperature sensor board, from Sensatronics(http://www.sensatronics.com/ [sensatronics.com]) I think. I think it was a 16 port board, and I bought all 300 foot cables for all of the sensors, and cut them to length. This ended up being about $5k I think(I went way overkill on the cable lengths).
At my current company we use servertech(http://www.servertech.com/ [servertech.com]) PDUs, their higher end models come with optional temperature/humidity sensors so we use those instead of the senatronics.
Despite it being a co-location, we had 500kW of power going into that cage(standard setup was ~12kW/rack), if the data center had followed their own procedures(AT&T enterprise network services), we would of had to have about a 5,500 sq foot cage, comparable to your data center
posting as AC, since I don't have an account. I read slashdot daily but I post maybe once every 2-3 years, so I haven't bothered to make an account.
ergotron = pricy but good (Score:2)
Re: (Score:2)
Not that power and cooling SOP is the only victim - we were down for a cage move this last weekend; most of the times we walked into t
crash cart (Score:2)
Requirementst of a Datacenter (Score:3, Interesting)
I bet you (Score:1)
Server Lift (Score:1)
I would suggest... (Score:5, Funny)
Monkey (Score:4, Funny)
Re: (Score:2, Funny)
speaking of which, I am willing to relocate for the right price.
I am guaranteed to:
Re: (Score:1)
Re: (Score:2)
*That* is engineering, I tell you.
Wireless Headset(s) (Score:4, Insightful)
Re: (Score:2)
Re: (Score:2)
Plantronics are great for the office, but suck in noisy environments. If you need to hear and be heard, there are alternatives.
A must... (Score:5, Funny)
Re: (Score:2)
Crash Cart? (Score:1)
-Peter
D Batteries (Score:3, Interesting)
The Tandem that houses our main clinical application had this big array of D batteries. We'd always made fun of the administrators because of it, but miraculously it stayed up when everyone else went down. I now bow down before their primitive greatness.
Re:D Batteries (Score:5, Informative)
I have a small site with about 8 computers and 3 servers and there is wireless shooting to 3 other buildings with about 4-6 computer each in them about 50 feet apart. I was over ridden in our battery backup system in favor of $50 ups purchased for each computer separately at office max. I'm thinking OK, they are getting a generator and I told them to make sure it had a line conditioner and was certified to work with sensitive computer equipment. besides, when it was just the one building, the UPS worked just fine.
They ignored that and one the test, after all the batteries went down, the computer just quit because the UPS software conflicted with a proprietary app the chose to use. I was called in by the guy who installed the generator and was told that about 20 of the UPS were bad. I though ok, they have been there for a couple of years in some cases and brought down some replacements. I swapped them out, they tested it again and before I got back to the shop I got a call saying more of them were bad. All the local sources were out and the electrician told me he had better backups so I told him to get them. after swapping them out I asked to make sure that they had a clean electrical line coming off the generator and they assured me there was.
Two weeks later, a car hits a telephone pole, the electric was out for more then 10 hours. All the UPS units went out, None of the computers would work. I tested the electrical line and it was jumping between 70 and 150 volts at about 40 hertz. All the ups shut down and wouldn't take power, they decided to plug the computers directly into the wall outlets and took two main boards out, three power supplies and the rest of the computers just wouldn't power up.
The data base on one of the applications got corrupted beyond repair and they had to recreate a weeks worth of entries because the drive got corrupted on the backup server too when the main board went out and no one had made the external backup in over 5 days. The phone system was borked, a 64 inch plasma TV in the lobby was gone, and various other things needed replaced because they acted weird from then on out. The line conditioners should have been about $90 per outlet or about $2000 for one capable of regulating all the power coming from the generator. In the end, it costs around $10,000 in replacements, labor and everything plus they ended up buying a new generator and this time getting a power control system that was certified for sensitive electronics.
Bad power will cause so many problems it isn't funny. Most people don't even know that a generator can be out of whack on output. Not all of them are created alike. Small things like how fast they can adjust to the load pulling from them and how stable the current is isn't a given. You have to make sure it is there or end up with broken electronic every time the power goes out and it kicks on.
Re: (Score:2)
How long did you expect the battery backups to last?
Decent tools (Score:2)
Plus, if you have decent tools, there's a chance people won't destroy them first chance they get and not replace them.
Re: (Score:2)
Rather, if you get decent tools they won't destroy them the first chance they get... they just won't return them
Re: (Score:2)
"Post 9/11" I took it with me to Bermuda, I made it through terminal security at 3 airports and gate security for two flights. The guy at the gate for my third flight said "we have to toss this", I was like "NO, you're not costing me $50 for a new screwdriver". I don't know why I wasn't gunned down for that (this was at Kennedy), but he let me on the plane. That's so not like me either, but I wasn't letting that thing g
Re: (Score:1)
WhenI saw the pink tool in his pants, I fired him! (Score:1)
We had a major problem with tools "walking" away when building our DC. Too many contractors, outsiders, and poorly-disciplined insiders. It was a MAJOR headache, with many hours lost hunting, deadlines missed, tempers and bad feelings, etc.
I went to the local big-box store and bought Plasti-Dip in five different colors. (Plasti-Dip is a liquid coating you dip the handles of tools into. It cures to a nice, thick, insulating layer, and makes for a good grip, too.) I assembled five complete sets of tool
Re:WhenI saw the pink tool in his pants, I fired h (Score:2)
Don't guess! (Score:5, Informative)
I've seen several good suggestions already with specific suggestions on tools or parts. Start with those. My suggestion is quite simple, actually: Why GUESS what you need, when you can find out for sure?
Tear down one ENTIRE rack. (Or several, if they have any variations.)
Now, look at this big pile of parts in front of you and imagine what you would do WHEN *ANY* one of them breaks.
Get several spares for each of those parts and put into the cart.
Whatever tools you needed for disassembly, put into a crash cart.
Then make another, identical cart. When the brown stuff hits the spinnie thingie, and multiple systems are down, the last thing you want to be doing is fighting over tools. Get spares of EVERYTHING so at least TWO people can work on things at the same time! You'll thank me when there's two of you trying to work on both sides of a rack.
NOTE: Be sure to inventory what you put into each cart! Tools have a way of growing legs and you want to be able to check and make sure that you STILL have ALL the tools.
And please consider getting a big-ass UPS for your cart (At least 1KVA). If your power is wonky, you want to be sure your cart's equipment (laptop, hub, switch, router, etc.) won't be flaking out as the power comes and goes. Even with the power out, you can plug one server into the UPS and restore/repair it while the power is still out. While you're at it, also get some LONG extension cords (100-foot) made of AT LEAST 12-gauge wire. Plug the UPS into the extension cord.
Think you're all set? Now, using ONLY the tools on ONE crash cart, put the rack back together. With the power out. (i.e. no mains)
When you have done this, not only will you be CERTAIN that you have all the tools you needed to [re]assemble everything, you'll actually have done so and will have run into (hopefully) most of the problems that you could encounter.
That's it off the top of my head. Best of luck to you! P.S. One last thing: MANY rolls of Duct Tape! <grin>
Re: (Score:3, Funny)
"ok, we've got every tool known to man, now hit the power - lets run the disaster recovery drill."
"umm... we did buy a torch, didn't we?...."
Re: (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
With all the good suggestions already out (Score:3, Funny)
This includes, but not limited to:
A bottle of whisky
A bottle of scotch
A glass
A Shotgun, pref with ammo
Sleeping pills
Pep pills....
In all seriousness, a good first aid kit should be in the center. Nothing sucks more than a dull headache and not having any asprin for it.
Plus, when someone cuts their hand on a server rack, it'll patch their hands up to keep them from bleeding all over them.
LARTs are always good ideas (Score:2)
Re: (Score:1)
Yeah, your datacenter could be key in coordinating the frantic research and resource distribution after an outbreak of the virus that reanimates dead flesh.
I'd suggest in addition:
Shoot... (Score:2)
- Maj. T.J. 'King' Kong (dec.)
Crash cart (Score:2)
http://www.eofficedirect.com/ProductDetails--895-- 23676--RUB9T28 [eofficedirect.com]
nothing worse than a headache when in a noisy room (Score:1)
Spares (Score:2, Informative)
Computer hardware isn't so much an issue - although, if you don't have some kind of maintenance contract, you want at least 2 of everything, up to and including 'entire servers'.
Depending on how much you're doing 'in house' things like cagenuts, spare cable management thingies, and tools to deploy said items will save a lot of grief.
Serial cables, and consoles, if you're running unix hardware. Get a set that you _know_ works. All too often you only ever need these when things hav
Guards (Score:1)
Re: (Score:1)
A few more things (Score:2)
Label maker. Preferably a decent one that can take different types of labels for different types of cables. Then go crazy with it. This will be a few hours of tedious work (but what are PFY, erm, Jr. Admins for?), and well worth it when you need it.
We have a large locking toolbox for our tools since they kept on walking. Funny, place is quite secure for servers, but screwdrivers ended up being left in pockets.
Talking about secure, if yo
toys (Score:2)
Don't forget the power and Mechanicals (Score:1)
- a drain snake and a drain cleaner. If you go to grainger, they have one that runs on co2 carts(like an air rifle uses). Condensation lines from air handlers will clog from time-to-time and the under-floor water detection system will alert you when you have a drain back up (you did get an under-floor water detection system?)
- a shop vac. to pull up any water should a drain back up.
- a multimeter with
A few more items (Score:3, Informative)
A Spot Cooler [airrover.com]- If you have a CRAC unit crap out and need some coverage while replacing part(s).
Replacement compressor(s) [carlylecompressor.com]
A variety of above floor fans, [grainger.com] and below floor fans [grainger.com] (in case of water under the floor).
As many spare breakers [geindustrial.com] as they'll let you buy. (that UPS is no good to you with a bad breaker downstream of it).
Don't just get tarps, get these tarps. [mauritzononline.com]
Extra long load bank cables [teledynereynolds.com]. Have your electrician make them up for you. If you make them extra long and store them onsite, you can use them to jumper out inside switchgear if you suffer a catastrophic failure (it might be ugly, but if done right, it can save your ass).
Flashlights [foreverflashlights.com] that will work.
Hand operated pumps [deanbennett.com]. If you have a pump fail and you need to get diesel fuel from your storage tanks to the "day tanks" of you generators, you'll be glad this is on the shelf.
A megger. [megger.com]
A phase rotation meter. [mygreenlee.com]
A good circuit tracer. [mygreenlee.com]
That's a pretty good start.
A big shop vac, water alarms, etc (Score:2)
Make sure you have water alarms under the floor as well. Ideally they notify someone directly when there's a problem.
Flashlights by the doors. Preferably rechargeable, and plugged in. But at least available,
Might be helpful (Score:3, Insightful)
Keep it current, keep it organized keep it available, and dammit Mike, put the pubs @$#%#@ back where they belong when you aren't using them!
Common hand tools in this box, commonly used special tools in that one. Rarely used tools in this other one. And dammit Mike, put the @$#%#@ tools back in the @$#%#@ box and put the @$#%#@ box where it @$#%#@ belongs when you aren't using them!
The main key is less in having lots of stuff than in keeping what you do have organized and available.
dont get duct tape get gaffers tape (Score:1)