Best Practices For Infrastructure Upgrade? 264
An anonymous reader writes "I was put in charge of an aging IT infrastructure that needs a serious overhaul. Current services include the usual suspects, i.e. www, ftp, email, dns, firewall, DHCP — and some more. In most cases, each service runs on its own hardware, some of them for the last seven years straight. The machines still can (mostly) handle the load that ~150 people in multiple offices put on them, but there's hardly any fallback if any of the services die or an office is disconnected. Now, as the hardware must be replaced, I'd like to buff things up a bit: distributed instances of services (at least one instance per office) and a fallback/load-balancing scheme (either to an instance in another office or a duplicated one within the same). Services running on virtualized servers hosted by a single reasonably-sized machine per office (plus one for testing and a spare) seem to recommend themselves. What's you experience with virtualization of services and implementing fallback/load-balancing schemes? What's Best Practice for an update like this? I'm interested in your success stories and anecdotes, but also pointers and (book) references. Thanks!"
Affordable SME Solution (Score:2, Interesting)
I am still in the process of upgrading a "legacy" infrastructure in a smaller (less than 50) office but I feel your pain.
First, it's not "tech sexy", but you've got to get the current infrastructure all written down (or typed up - but then you have to burn to cd just in case your "upgrade" breaks everything).
You should also "interview" users (preferrably by email but sometimes if you need an answer you have to just call them or... face to face even...) to find out what services they use - you might be surprised to find something that you didn't even know your Dept was responsible for (oh, that Panasonic PBX that runs the whole phone system is in the locked closet they forgot to tell you about...)
Your next step is prioritizing what you actually need/want to do... remember that you're in a business environment so having redundant power supplies for the dedicated cd burning computer may not actually improve your workplace (but yes, it might be cool to have an automated coffee maker that can run on solar power...)
So now that you know pretty much what you have and what you want to change...
Technology wise, Virtualization is definitely your answer... and there's a learning curve:
VMWare is pretty nice and pretty expensive.
Virtualbox (I use) is free but doesn't have as many enterprise features (automatic failover)
Xen with Remus or HA is the thinking man's setup
All of the above will depend on reliable hardware - that means at least RAID 1, and yes you can go with SAN but be aware that it's a level of complexity you might not need (for FTP, DNS, etc.)
Reading what you've listed as "services" it almost sounds like you want a single linux VM running all of those things with Xen and Remus...
Good luck, and TEST IT before you deploy it as a production setup.
P2V and consolidate (Score:5, Interesting)
Re:Trying to make your mark, eh? (Score:5, Interesting)
I, personally, am TOTALLY in agreement with the ethos of whoever designed it, a single box for each service.
...
Virtualisation is, IMHO, *totally* inappropriate for 99% of cases where it is used, ditto *cloud* computing.
I totally disagree.
Look at some of the services he listed: DNS and DHCP.
You literally can't buy a server these days with less than 2 cores, and getting less than 4 is a challenge. That kind of computing power is overkill for such basic services, so it makes perfect sense to partition a single high-powered box to better utilize it. There is no need to give up redundancy either, you can buy two boxes, and have every key services duplicated between them. Buying two boxes per service on the other hand is insane, especially services like DHCP, which in an environment like that might have to respond to a packet once an hour.
Even the other listed services probably cause negligible load. Most web servers sit there at 0.1% load most of the time, ditto with ftp, which tends to see only sporadic use.
I think you'll find that the exact opposite of your quote is true: for 99% of corporate environments where virtualization is used, it is appropriate. In fact, it's under-used. Most places could save a lot of money by virtualizing more.
I'm guessing you work for an organization where money grows on trees, and you can 'design' whatever the hell you want, and you get the budget for it, no matter how wasteful, right?
Re:Trying to make your mark, eh? (Score:4, Interesting)
Get real, for 150 users at WRT54 will do DNS etc....
Want a bit more poke, VIA EPIA + small flash disk.
"buy a server".. jeez, you work for IBM sales dept?
Maybe this is really a uni project (Score:3, Interesting)
If the current system has been acceptable for 7 years, I'm guessing the users needs aren't something so mindbogglingly critical that risk must be removed at any cost. Equally, if that was the case, the business would be either bringing in an experienced team or writing a blank cheque to an external party, not giving it to the guy who changes passwords and has spent the last week putting together a jigsaw of every enterprise option out there, and getting an "n+1" tattoo inside his eyelids.
Finally, 7 years isn't exactly old. We've got a subsidiary company of just that size (150 users, 10 branches) running on Proliant 1600/2500/5500 gear (ie 90's) which we consider capable for the job, which includes Oracle 8, Citrix MF plus a dozen or so more apps and users on current hardware. We have the occasional hardware fault which a maintenance provider can address same day, bill us at ad-hoc rates yet we still see only a couple of thousand dollars a year in maintenance leaving us content that this old junk is still appropriate no matter which we we look at it.
Re:Cloud Computing(TM) (Score:4, Interesting)
I think what the article is really asking is what's a good model to start all this stuff. You're looking at one or two servers per location (or maybe even network appliances at remote sites).
I totally agree with your premise. In my experience taking something that appears to work (when you realize you've really just been lucky) requires some time to bring about the change that the business really needs.
Now, as for having two servers per location, that heavily depends on how those sites are connected. Are they using a dedicated line or a VPN? That's important since that'll affect what hardware needs to be located where. It's possible (even if unlikely) that some sites would only need a VPN appliance... But since the poster seems to want general advice:
VMWare ESXi is a pretty good starting place for getting going on virtualization. I've had a great experience with it for testing. When you feel like you've got a good handle, get the ESX licenses.
If SAN isn't in your budget, I still recommend some sort of external storage for the critical stuff... Preferably replicated to another site... But you can run the OS on local storage, especially in the early stages. But you'll need to get everything onto external storage to implement the VMotion services and instant failover. Get a good feel for P2V conversion. It'll save you tons of time when it works... It doesn't always, but that's why you'll always test, test and test.
As for the basic services you stated above (www, ftp, email, dns, firewall, dhcp):
Firewall (IMHO) is best done on appliance. Which should be anywhere you have an internet connection coming in. I'm sure you knew that already, but I'm trying to be thorough.
Email is usually going to be on its own instance (guest, cluster, whatever)... But I find that including it in the virtualization strategy has been quite alright. In fact, my experience with virtualization has been quite good except when there is a specific hardware requirement for an application (a custom card, or something like that). USB has been much less of a headcache since VMWare has support for it now, but there are also network based USB adapters (example: USBAnywhere) that provide a port for guest OSes in case you don't use VMWare.
Re:Trying to make your mark, eh? (Score:3, Interesting)
Years ago the Microsoft DNS implementation had a very nasty memory leak and used a lot of cpu - you really did need a dedicated DNS machine for small sites and to reboot it once a week.
I think that's why people are still thinking about putting it in a virtual box so it can't eat all the resources, even for a pile of trivial services that a sparcstation 5 could handle at low load.
In practice, everyone just builds two domain controllers, where each one runs Active Directory, DNS, DHCP, WINS, and maybe a few other related minor services like a certificate authority, PXE boot, and the DFS root.
I haven't seen any significant interoperability problems with that setup anywhere for many years.
Still, virtualization has its place, because services like AD have special disaster recovery requirements. It's a huge mistake to put AD on the OS instance as a file server or a database, because they need to be recovered completely differently. The last thing you want to be doing during a restore is juggling conflicting restore methods and requirements!
Most of the poster don't 'get it' (Score:3, Interesting)
The question is not about hardware or configuration. It is about best practices. This is a higher level process question. Not an implementation question.
Re:Trying to make your mark, eh? (Score:3, Interesting)
No, you need seperate servers for when the DHCP upgrade requires a conflicting library with the DNS servers which you don't want to upgrade at the same time.
THIS is where virtualization becomes useful.
On the other hand, my solutions is a couple of FreeBSD boxes with jails for each service. You could do the same with whatever the Linux equivalent is, or Solaris zones if you want. No need to actually run VMs.
Just run a couple boxes, seperate the services onto different jails. When you need to upgrade the core OS, do it on your backup box first, get all the services upgraded, switch it to your primary and repeat on the other.
Its not a matter of config files, its a matter of dependencies. If you've never run into a dependency conflict, you don't have much experience. Upgrading every service at the same time isn't always an option, sometimes newer versions in repositories are broken with regards to something you use or need.
Re:Think about the complexity of duplication (Score:2, Interesting)
Maybe this is asked elsewhere in these threads, but the one thing that seems to not be asked here is not just "What are the business requirements?", but also "What are your business application requirements?". While it may seem implied in the former question, IME, it is usually not addressed enough by simply asking the former. In asking the former, it seems that you get nice "businessy" answers like "we need Y application to be back up and running in X time". What it doesn't answer, is what are the requirements for Y application? Does it need to have internet connectivity, connectivity to a central database, or is it completely stand-alone? In the second case, unless you have a sufficiently advanced application (most aren't), simply putting an instance of Y application locally in case your link goes down, may not cut it if it does not have suitable "caching" mechanism to store data until the link comes back and then forward it on to the central DB.
I have seen many hardware upgrades "fail" even though the upgrade was technically successful. This was usually caused by the project team asking the right business questions, but forgetting to drill down and ask the right questions of the application providers (vendors or internal development staff).
I was actually involved in a Active Directory "upgrade" project where the project team was wanting not to simply upgrade AD to the latest version, but also refactor the directory structure (due to some really poor choices on the initial implementation which was causing daily grief for the maintainers of the information), without considering the impacts to the applications we had built in-house that were using AD for Authn and authz (most would've likely been able to handle the changes since they were fairly configurable in this regard). I raised this concern many times and almost everytime, it was ignored, or it was "yeah, we will consider that", and then it got dropped on the ground. Fortunately, just prior to implementation, the project got "put on the back-burner" and the project manager (a contractor) was let go due to "budget cuts". Hopefully when/if this gets traction again, we will actually look at what else besides the network and people's workstation login's will be affected.
I still struggle to understand what causes this rift between infrastructure people and development people (I have been on both sides, but mostly on the development side), as a poor application choice can severely restrict what can be done with a company's hardware, and inversely, a poor infrastructure choice can unexpectedly break an application.
However, if you are only a company of 150ish employees, hopefully you are still small enough to deal with issue quickly and efficiently (it seems to get worse as corporations get bigger).