Servers - To Colocate Or Not To Colocate? 13
BilldaCat writes: "I would like to know which is the better choice for a start-up: colocating, outsourcing or bringing the servers in house. Office space is not an issue, as it is already paid for. Money is not so much of an issue, but if it's possible to save $ without losing reliability/service, that would be what we want to do. Flexibility and growth is key. We expect a fairly quick ramp-up period in a couple months. Personally, my bias is to bring the stuff in house, as reliability is very important to me, and I feel with some of the larger companies we may not be treated as well as I would like, since we plan on starting with only a handful of servers. If anyone has any experience in these situations, I would really appreciate some feedback!"
Re:Depends on your wants and needs... (Score:1)
make that 3!!!!!
(fortunately we just moved, so this 3rd one won't hit us)
lesson learned: get a written guarantee, that there will be no power downtimes
greetings, eMBee.
--
Don't bother hosting yourself (Score:1)
Re:Depends on your wants and needs... (Score:1)
*Not a Sermon, Just a Thought
*/
Wherever you go, get design assistance... (Score:1)
1) Decide on pure colo vs. managed colo. Some companies will manage your boxes for you if you buy the hardware from them and pay their monthly management fees.
a) You get backup included, and someone's always watching out for your box... getting paged at 3am because some log filled up your drive -- then having to get up, connect in, and fix it -- is a downer.
b) You don't have root. If you are using non-supported apps, this is often unacceptable.
2) Get someone who does hosting to assist with your design. Don't hesitate to pay a few grand up front.
a) Many hosting companies have experience with load balancing, HA, and network security. Your techies may be great with development, databases, optimization, and the like, but most larger hosting companies deal with financial institutions on a regular basis. They fully understand high availability in every sense.
b) It's wonderful that you've got a great site. Have you even considered an SSL accelerator, for example? An nCipher nFast, for example, can dramatically increase the number of SSL connections your server can make per second, and dramatically lower CPU overhead. Hosting companies know about things like this, which your techies may not. They also know, for example, that the reason that it appears that the iowait on the box with a SCSI-connected nFast is high is simply because the device keeps open a channel to report back errors from the device to the driver, and is not actually seriously affecting your box like it may seem. This kind of troubleshooting panic can be avoided by discussing your plans with someone who "does hosting".
When you look at the expense of hiring 24x7 staff to look after your boxes, it's often worth the cost of paying for a managed service.
-Nev
The hardware you choose can help alot or hinder. (Score:2)
Choose your server hardware carefully. Many companies are now offering access to the PC keyboard and BIOS via a second NIC with dedicated hardware. A means to power cycle the machine also helps.
Design your system for redundancy. Possibly add a spare system or two with all software, data loaded. This way it can take over the work of a failed system. This spare system could also be loaded with a couple of huge drives to be a backup host for the other servers. One site I know of has a management system that also provides backup service, and can be used as a spare server too. They have a firewall/load ballancer, 5 web servers, 2 database servers, and the management system in 10U of space. The tenth slot is the keyboard/flatpannel display tray for the management system. All the systems are Linux based with serial consoles. The management system can also power cycle any of the other systems and it's self.
Re:Depends on your wants and needs... (Score:2)
We expect on starting out with a T1/T3, then ramping up to whatever bandwidth we need within a 4 month period.. either the idea is going to take off or not at that point. Downtime is very "not good". I've already stressed the need for a UPS, but you bring up a good point about the blackout situation/generator.. hmm. Interesting.
I know going with a non-reputable co-lo, or finding one of the cheap co-lo's is not going to cut it.. if we go co-lo, we need something that is going to be up all the time, a way to handle server reboots when needed, and easily scalable. I'm located in Northern VA, so if anyone knows anyone in this area, I would appreciate it.. AFAIC, the closer the better, in case I ever need to go in there.
Don't forget fire supression... (Score:2)
Most office-type sprinkler systems are likely going to be water-based...
I'd dispute that having someone on-hand 24/7 is as big a need as you think, with a properly designed network. In a proper network you eliminate *all* single points of failure, and a non-redundant router/firewall/load balancer (or one that has to be manually switched over) is BAD BAD BAD. With the right equipment, you can even remotely power-cycle a hosed machine.
It's all about how much you want to spend really...
Also, to add my $0.02 on the power issue: The colo my daytime employer uses has seperate connections to two different power grids. Then, they have an entire *floor* filled with battery banks to provide 2-3 hours of backup. The batteries feed into line conditioners before the power ever gets to the servers. If the batteries start to get low, there's a 1,000 gallon hot-fillable generator in the basement which can feed the batteries for 24 hours before it needs more fuel.
And on the redundant links issue: This gets one more advantage -- growth. The lead time on a T1 is about a month. I'm sure a T3 is at least that. If you start to experience sudden growth, you want room *now*. Most colo's wont exceed 50% capacity before they add even more fiber, and you're talking upwards of 30Gb to begin with.
-JF
Re:Depends on your wants and needs... (Score:2)
our office building had 2 sceduled power downtimes for a full day!!!!!
very annoying, i tell you. (our stuff is colocated fortunately)
most officebuildings simply are not managed with 24/7 uptime requirements in mind. we were just one of the smaller offices, and i can't see how we could have demanded that they simply never power down...
since i think it's pretty much impossible to provide your own power in case of longer outages, your only option is to ask whoever you are leasing from, if they guarantee full uptime, and provide backup strategies, or just use a colo
greetings, eMBee.
--
watch out for SLA's and contract terms!!! (Score:2)
IF you are thinking that you are going to be changing / growing fast within the next 4 months watch out for your decision. THe process of getting an agreement with one of the good places (Exodus/Navisite etc) can take a bit and you want to be able to get the hardware setup etc.
A few things you have to think off : ARE YOU IN THE bay area... if you are forget about doing it yourself. you will need to get new ppl. fast and have them working fast, this cannot happen. I have completely given up on the idea of being able to find qualified ppl. even for a decent salary in the bay area (recruiters fees and other costs make colocation well worth it).
Even if you think you will be able to pull in all the ppl. you need, you have to think of the other problems with you being a smaller place now, getting additional power/cooling is a big planning job, once in a contract you can get more rack space easy!
Either way, be careful with the contract... that is the main advice i can give you.
Depends on your wants and needs... (Score:3)
It really depends on your requirements. How much downtime can you stand? How much traffic are you going to get and how fast will it grow? Is it cost effective for you to deal with the infrastructure issues yourself or should you let the coloc people worry about it?
Bandwidth, power and cooling are going to be your main issues. Let's talk bandwidth first. How fast is your traffic going to grow? Major colocs like Exodus [exodus.com] have plenty of spare bandwidth on hand to sell you. Telcos can take weeks to run a new line. If you do it yourself you have to really keep on top of your line utilization or you can find yourself way behind the curve and it's a bitch to try and catch back up.
As far a cooling goes, you're probably going to be fine unless you have to add a ton of servers. But this is one thing that's bitten me in the past. We outgrew our cooling capacity in just a few months and we paid the price for months until we got another unit installed.
And let's not forget the power issue. How much downtime can you afford? A good coloc is going to have a generator as well as a good UPS system. Can you stand your site being down 4 hours because of a blackout?
The other benefits of a good coloc is their experience and full time staffing. I've very rarely seen the good ones have anything but minor network outages (backhoes not withstanding. :) Their NOCs often catch problems long before you hear anything from your users.
The major downside to a coloc is the expense. The good ones cost a ton of money. The only way to cut down the expense is to make sure you go rackmount and use the minium Us you need to do the job. They'll also hit you up for bandwidth costs, but that's gonna directly relate to your traffic so it's less of an issue.
So it all pretty much boils down to balancing downtime and headaches vs. costs. And that's math only you can do.
Re:Don't bother hosting yourself (Score:3)
Particularly when you're dealing with an Akamai situation, you need to be physically distributed, and that's what they're getting by being all over the place.
Think about bandwidth costs (Score:3)
When you buy bandwidth at a colocation facility, you are buying a 100mbps ethernet port (cheap), plus a share of an OC3 (or an OC12 or whatever).
When you buy bandwidth to your offices, you are buying a T1/fractional T3/T3/whatever. Economies of scale come in here: at least from AT&T, an OC12 connection is 4 times cheaper than a T1 on a per-gigabyte basis.
Unless you are going to be using a huge amount of bandwidth, it is going to be cheaper to share a fat pipe with other people than to buy a narrow pipe for your self.
It's a question of reliability. (Score:4)
So with that in mind, the real issue is one of reliability. Consider the following factors:
- Cooling. Yes, it's been said, but it's very important. I'm at a software company, and even if we were to have the ability to control our cooling needs (and we don't at the moment, which is a major limiting factor), you'd be amazed at the kind of heat you'll put out. I've seen rooms where the cooling failed, and it's not pretty. Colo facilities have cooling situations that would make you drool, and they're redundant.
- Redundant Links. For the price of one link, at a competant colo facility, you get several links to several backbones. One goes down, you're fine. Two go down, you're still fine. Even if you get dual T-1s or dual T-3s on your own, unless you actually lay your own fiber, you're probably dealing with your local RBOC owning the cable itself. So there's still a central point of failure. If the problem is with Sprint and you have an MCI link, you're fine. But if the problem is with SBC/PacBell, you lose regardles of how redundant the links you have to your office are. People often forget this.
- Security. Ever had a janitor accidentally pull out a cable of any kind? I have. That won't happen at a colo facility.
- Someone's there 24-7. At the colo we're using, let's say we get a page at 3:30am which says that our firewall has died. I can actually call the colo staff, and have one of them walk to our rack, pull ether from one box and into another, and go back to sleep. This is actually a big advantage.
- Power redundancy. This is another big issue. Let's say you have a power outage in your office. Fine, your servers are on a UPS. Is your phone company's routers? Who knows. Let's say the outage lasts a while. Then everyone loses. Unless you spend a fortune figuring out everything in your setup which is power dependant (don't forget the A/C! If you have a power outage and your A/C goes, you might start burning CPUs in a matter of an hour or so), and making it redundant, what about one block away where your phone company is?
In short, the real issue with a colo is truly reliability. You could spend all your time and money setting up a comparable situation to a colo facility in your own office, but why bother? It'll just cost you more, and those are sunk costs which your CFO won't be happy about.