UPS Setup For a Small/Mid-Size Company? 260
An anonymous reader writes "We're a small company employing ~30 people and we are becoming increasingly reliant on virtual servers. Unfortunately, the hosts they are on don't have redundant power supplies because we simply don't have the capacity. We currently have one UPS per rack, which gives us about two minutes. This may have been enough time when they were put in — they've been there for some time — but it isn't really enough time to shut everything down in the event of a failure. Domain Controllers alone may take up to 15 minutes. So I'm looking at upgrading the UPSs to ones that would preferably give us around 15 minutes of breathing space and send an email or text alert when a failure is detected. Something that could trigger shutdowns automatically would also be nice. Of course cost is a key factor too. so given all of the above, what does Slashdot recommend?"
Have you tested the UPS lately? (Score:5, Insightful)
This is sort of off topic, but when was the last time you tested the UPS units that were installed "some time ago". The batteries can eventually go flat. You better check what you have ASAP. You may need to replace them sooner than you think.
I can't remember the brand, but some of the higher end UPS units I have used came with monitoring software. They software polled the UPS unit, and started the shutdown as soon as a power failure caused the switch over to battery.
HTH.
A second site (Score:3, Insightful)
With redundant connection.
1st. Identify Requirements (Score:5, Insightful)
Diesel (Score:5, Insightful)
No matter how much battery capacity you have, it will eventually run out. If your site truly needs availability, you have to get a diesel generator.
Re:stationary bikes with alternator (Score:3, Insightful)
Cost of UPS vs Colo? (Score:3, Insightful)
What's the cost of a good set of UPSes vs simply migrating to a Colo & fatter pipes? Datacenters (most of them anyways) promise at least a few hours of generator uptime, and it sounds like you're already using a colo somewhere (dns relocation, etc).
Re:Have you tested the UPS lately? (Score:3, Insightful)
Re:Diesel (Score:5, Insightful)
If your site truly needs availability, you have to get a diesel generator.
lol.
If your site truly needs availability, you need a second site.
Re:Have you tested the UPS lately? (Score:4, Insightful)
If you have a server with redundant power supplies- then you can have each power supply attached to a different UPS, then you can test each UPS one by one, hopefully without the server going down due to one UPS failing the battery test.
You don't necessarily want to shutdown immediately. I have my machine shutdown once the software thinks there's only X seconds of battery life left. Set X to something high enough so that there's enough time to shutdown AND cold boot AND shutdown again... Otherwise in event of a shutdown you will have to wait some hours till the UPS is charged enough before it is safe to power up - in case the power company or whoever cuts power on you half way during your boot up
Leave it to the professionals (Score:5, Insightful)
Co-locate your equipment at a carrier-grade data center in the nearest major city to your location and get a leased line to your premises. A decent data center will have proper battery backup and generators and know how to handle it. They'll also have the time and manpower to do proper tests, etc.
Re:1st. Identify Requirements (Score:5, Insightful)
You're absolutely right. One place I worked had about 20 employees, 150 servers, but had an income of millions per year. The income averaged out to about $5,700/hr. 12 hours of outages per year could cost almost $70,000 in lost revenue. Is it worth $10k in extra equipment to mitigate that? Obviously.
Smaller companies have to evaluate their acceptable losses. Sometimes it's not worth $100 to make sure you stay up through power outages.
"5 9's" of reliability still leaves 1.14 hours per year of outages. Of course, that doesn't assume that it's all power related outages. Redundancy across physically diverse locations can and will help there.
Re:Diesel (Score:4, Insightful)
Protection against backhoes is a redundant path, not a redundant site.
Re:Have you tested the UPS lately? (Score:3, Insightful)
The best way to test your power backup system is to throw the main switch and see what fails.
I'd say that's the most rigorous, realistic way to test, but I'm not sure that it's the best way to test.
So, you pulled the main breaker and took out all of the production servers because there was a problem with your UPS configuration software? Oops. Why didn't you try it on one server today, then one rack tomorrow, and then pull the plug on the whole system after close on business on Friday?
Pulling the Big Red Switch is one good way to test, but it's not the only way, and it almost certainly shouldn't be first thing that you try.
Re:Have you tested the UPS lately? (Score:3, Insightful)
What if the power goes out on Wednesday?
That's my point, really. Test in situations where a failure has less severe consequences and you can troubleshoot smaller pieces first. Do the full-on Wednesday-morning test after you can pass the other tests.
While it's important to be able to recover from a power failure, it's also important that designing and testing your redundant power solution doesn't do more damage to the business than not testing.
Re:Have you tested the UPS lately? (Score:3, Insightful)
The power going out on Wednesday is why you run your test on Friday--you test your protection to ensure that you can recover from a mid-week power failure by Thursday morning, but you don't run the actual test on Wednesday. You set your test up so that a catastrophic failure causes the least disruption.
When I was in the Navy, we would run drills in the same manner--our response was the same whether the sub was running deep and fast or slow and shallow, so why run the drills in the riskier situation of deep and fast, where a catastrophic failure in response to the drill could cause very bad problems very quickly?
But in a data centre you don't face the risk of drowning and/or perishing. Also, it's really easy to convince yourself that your setup works because you've carefully and methodically powered down each backup source individually, but it's near impossible to determine the stress that would be placed on a network by a power failure unless you simulate one. Obviously you have to take steps before hand to make sure your stuff is reliable and you're not going to run a full shutdown while everyone is accessing the data (that, by the way, is a lovely strawman everyone is putting up).
The fact of the matter is, if you're not confident enough in your data centre's reliability to throw the switch, your data centre just isn't reliable enough.
Re:Have you tested the UPS lately? (Score:3, Insightful)
The fact of the matter is, if you're not confident enough in your data centre's reliability to throw the switch, your data centre just isn't reliable enough.
Well, yeah. That's really the whole point in this discussion - they don't know it's reliable enough. Therefore, throwing the switch on the whole center at this point is foolish at best. You don't do that until you're reasonably confident based on other smaller tests you've done first. Which I believe was exactly what was being stated. Sometimes you have to learn to read between the lines, and understand things without having every little detail explicitly spelled out for you.
Re:Inverters (Score:1, Insightful)
The reason your lead-acid battery solution runs so long is that you have A-h (amp-hours) to back it up. If you look in an APC UPS, you will find one or more sealed-lead-acid (SLA) batteries. The ones I've seen are 7 A-h batteries. Say your load is 1000 VA, how long are those 2 little 12V, 7 A-h batteries going to hold up? Do some math, convert the 12V, 7A-h to VA-h: 2 * 12V * 7A-h = 168 VA-h. Divide this number by your load: 168 VA-h / 1000 VA = 0.168 h, or about 10 minutes, optimistically.
Slap a couple of [honking] A-h car batteries behind an inverter, and you easily have lots of run-time. APC and other UPS manufacturers do not seem to want to tell you VA-h capacity of their units, but you can figure it yourself by knowing the capacity of the batteries.
One disadvantage of car batteries is the hydrogen and acidic gases given off by the batteries. They have to be in a separate room from the servers.