Power Distribution in a Datacenter? 33
d00dman asks: "I work for a colocation reseller. We don't own our datacenter, we lease cage space and with it we get an array of 30amp and 15amp power circuits. My problem is, inside of the cage we need to do our own power distribution. We have around 50 hosts in the facility, and are growing quickly. We need to add another 20 or so hosts in coming month. We are looking for a simple, scalable, and even power distribution method. As always, money IS an object. Right now we have a group of UPS's with power strips plugged into each. While this was sufficient for as many as 30 hosts, its starting to become a bit of a mess. Does anyone have any advice to offer...any creative tricks?"
Of course you've already thought of this... (Score:5, Insightful)
Many datacenters now provide fully conditioned, bus-sized UPS-backed power. Some even let you control individual power outlets from an SSL/password-protected web interface (Which is extremely useful if/when a server hard locks). Even if the datacenter doesn't provide the web-based outlet management, you can pick up a 1u, 8-outlet APC power switch for $300. 5 of them would provide power for the whole rack (Assuming single power supply), and only cost $500.
You've got my vote... (Score:2)
Can you let us know more details that you weren't able to ask in the original question?
Re:You've got my vote... (Score:2)
A power conditioner insures that the electricity you are using is clean, fairly glitch free, smooth power. Protects you from power spikes or hot power (like sustained voltages over 120v on your 110v line). When there is no power (power outage) you get a nice smooth glitch free zero volts.
A UPS is just a quick switch over from wall power to a battery. When there is no power the UPS uses an inverter to give you 110V AC from batteries, generally lead cell sealed batterys, I am guessing two six volt batteries hooked up in series to give the UPS 12V DC to start with.
Re:You've got my vote... (Score:3, Insightful)
Re:You've got my vote... (Score:3, Insightful)
Usually people plug small systems in for the wrong reasons... 'what if this huge building system that is maintained regularly "breaks?" Good think I have this little box here that I've been using for the past five years without any problems..!' or 'I want to be able to move the server around the room without unplugging it' nonsense.
There are very real problems associated with plugging the little shits into a large system; the cheap inverters have trouble tracking and can create all kinds of problems.
Also, always remember that the sealed batteries only last 3-5 years if you are lucky!
Re:Of course you've already thought of this... (Score:1)
Re:Of course you've already thought of this... (Score:3, Informative)
In fact, I solicited bids for housing ~300 servers a couple years ago from 12 different data centers and EVERY SINGLE ONE had backup power. These include Frontier Global center (before Exodus bought them) Exodus, MCI, Level 3, Globix, Equinix, AT&T, Best, Netcom, AboveNet, GTE, the list goes on.
If your datacenter provides such shit power that you need your own UPS, DUMP THEM NOW. They SUCK. You are NOT getting your money's worth. FWIW, I've also built 6 data centers. You REALLY don't want a bunch of little UPS's all over the place. They aren't very good anyway. Depending on your uptime requirements, get your self one (or two) BIG UPS system. They come in all sizes. Best has some that go to 220kVA even. I bet a 15kVA would do ya though.
Call an electrician (Score:1)
You could also call a vendor like APC [apcc.com] and ask about power distribution to x00 rackmounted hosts, I'm sure they have a solution they will sell/lease you.
APC & Baytech (Score:3, Informative)
Doing this myself actually.
Call the electric company (Score:2)
Seriously, just call up UI or whoever does the electricity in your location. Tell them you need better, and more power. And get a single backup generator, rather than many UPS'. Sell all the UPS and just have the electricians install many power outlets where necessary. You'll have a few minutes of downtime will you unplug/replug, but that's about it.
Do you really need your own UPS's? (Score:2)
What's wrong with the data center's UPS's? By definition, if it's really a data center then it's gotta have UPS's, right?
You do *not* get better reliability by putting two UPS's in series. You do get better reliability by getting rid of unreliable UPS's.
Re:Do you really need your own UPS's? (Score:1)
Enough ranting for today, i swear.
Power Recommendations (Score:5, Insightful)
First off, get rid of the stand-alone UPS systems. They are inefficient and not reliable from a system standpoint. You are much better off with a single 30kW unit rather than ten or twenty 1,000VA units. With single UPS, it is easy to either install a local panelboard (US), or a busbar distribution system (UK).
Second, anything that you do to enhance flexibility will compromise reliability. If that is an acceptable risk they go ahead and do something modular.
Given these constraints, a 30kW UPS is 100A or so three phase, 208V or 150A single-phase 208V. 120V would not be appropriate; 480V is a cheaper input feeder, more hassle for bypass.
The best solution might be to go the traditional panelboard route, and run branch circuits to all your equipment in liquidtight flexible non-metalic conduit (the blue stuff).
Alternately, you can install a 100A plug-in busbar the length of your cage and tap off 30A, 208V, 1PH for each rack. They are designed for light industrial applications, but work pretty well if you can sacrifice on reliability.
The APC solution [apc.com] is neat because you don't need an electrician, and it is all modular. Might not be as flexible as the component route, but it works. I think it includes managed outlets in the racks as well.
Re:Power Recommendations (Score:1)
You may be right about the efficiency, and probability says you are right about the reliability, but I'd disagree with the statement that you are "much better off" with One Big UPS as opposed to Many Little UPSes. Yes, UPSes in the ~30kW range are typically designed/engineered better, provide "better" power output, are generally less likely to die in the arse than a UPS in the ~2kW range, and more likely to fail over onto bypass rather than just stop providing power altogether, however, replacing Many Little UPSes with One Large UPS does create a single point of failure.
In the current setup, if any one UPS dies, it affects only the equipment attached to that one UPS. With One Large UPS, if it dies, all the protected equipment dies. If reliability was a concern (as I imagine it is) I'd be much happier with 2* 30kW UPSes, or 3* 15kW UPSes.
Also, check the electrical codes in your jurisdiction. I dont know the exact situation in the US, but in Australia recent legislative reforms (caused by too many brain-dead sparkys frying themselves) mean that performing "live work" (ie tapping live busbars etc) now comes with a *significant* amount of administrative BS (significant enough that a lot of sparkys who follow the rules will no longer do live work. Of course, the brain-dead ones will continue to fry themselves). I've had electrical fitters from a major UPS vendor tell me that they're are no longer permitted (by management) to perform live work for any reason, so extensions to an existing system will require the complete shutdown of the protected power (and all the equipment attached to it)!
42 V DC busbars (Score:5, Informative)
- Convert incoming power (whatever form) to 42VDC.
- Have a big bank of batteries charged by the 42 VDC.
- Run all your equipment from power cables attached to big copper busbars with 42V between them.
Advantages: you get rid of your UPS's, you have a very scalable power system (telcos never unplug anything!), lots of big manly metal things.Disadvantage: dropping a wrench across the 42V busbars is a bad thing
If you don't have access to a lot of surplus rackmount PC's with 42VDC-in power supplies and are really on a budget, do it with 12VDC instead and use these servers that run off 12V very nicely [mini-itx.com].
Economical solution (Score:2, Informative)
Use a PDU (Score:2, Informative)
http://www.google.com/search?hl=en&ie=ISO-8859-1&
I personally would not buy from APC. I've had more server failures due to APC UPSs shutting off the power output than I would have had with power failures. Our power company is more dependable than APC. Small UPSs are banned here.
servertech makes what you need (Score:3, Informative)
Power raceways (Score:1)
The solution we are planning on using is some neat power racewys from Universal Electric Corp. Here's the link: http://www.uecorp.com/starline/starline.htm [uecorp.com]
These are pretty slick--the tracks are fixed in place, but the outlets are completely mobile. Think track lighting, but for power outlets. The modules come in all the various plug types ranging from standard 110v 15A plugs to 208v plugs to bigger modules with fuseboxes and circuit breakers (the list of plugs is pretty extensive).
The major benefit for this raceway is that once the track is installed, the plugs can be relocated without shutting down power, or requiring an electrician to come in and do it. You can also buy modules as needed, instead of having to guess at options for future expansion. In our case, we are going to get several 110v/20A modules and a few 208v outlets, place them directly over the racks as needed, and have power strips internal to the rack plug into those.