Forgot your password?
typodupeerror
Networking IT

Best Practices For Infrastructure Upgrade? 264

Posted by timothy
from the thinking-ahead dept.
An anonymous reader writes "I was put in charge of an aging IT infrastructure that needs a serious overhaul. Current services include the usual suspects, i.e. www, ftp, email, dns, firewall, DHCP — and some more. In most cases, each service runs on its own hardware, some of them for the last seven years straight. The machines still can (mostly) handle the load that ~150 people in multiple offices put on them, but there's hardly any fallback if any of the services die or an office is disconnected. Now, as the hardware must be replaced, I'd like to buff things up a bit: distributed instances of services (at least one instance per office) and a fallback/load-balancing scheme (either to an instance in another office or a duplicated one within the same). Services running on virtualized servers hosted by a single reasonably-sized machine per office (plus one for testing and a spare) seem to recommend themselves. What's you experience with virtualization of services and implementing fallback/load-balancing schemes? What's Best Practice for an update like this? I'm interested in your success stories and anecdotes, but also pointers and (book) references. Thanks!"
This discussion has been archived. No new comments can be posted.

Best Practices For Infrastructure Upgrade?

Comments Filter:
  • I've been looking at hp c3000 chassis office-size blade servers, which may serve as your production+backup+testing setup, and scale up moderately for what you need. Compact, easily manageable remotely, and if you're good about looking around, not terribly overpriced. Identical blades make a nice starting point for hosting identical VM images.

    • Blade servers are very nice for more than, say, 8 servers purchased at a time. The built-in remote integration of better blade servers, the trivial wiring, and physical management are sweet. But the blade server itself becomes a single point of failure, much as a network switch can be, so it takes thought to install and manage them properly. And they cost, at last glance, roughly $500/blade for the chassis. Is this worth an extra $500/server on your budget? Not if your servers are quite modest and the perso

    • Re: (Score:3, Informative)

      by Z00L00K (682162)

      Any server that can offer a RAID disk solution would be fine. Blade servers seems to be an overkill for most solutions - and they are expensive.

      And then run DFS (Distributed File System) or similar to have replication between sites for the data. This will make things easier. And if you have a well working replication you can have the backup system located at the head office and don't have to worry about running around swapping tapes at the local branch offices.

      Some companies tends to centralize email around

    • In my uninformed opinion, blades are mainly a way for hardware vendors to extract more money from suckers.

      They probably have niche uses. But when you get to the details they're not so great. Yes the HP iLO stuff is cool etc... When it works.

      Many of the HP blades don't come with optical drives. You have to mount CD/DVD images via the blade software. Which seemed to only work reliably on IE6 on XP. OK so maybe we should have tried it with more browsers, than IE8, but who has time? Especially see below why you
    • The biggest problem I've found with blades is that you can't fill a rack with them. Several of the datacenters I've come across have been unable to fit more than one bladecenter per rack. Cooling and power being the problem.

      At the moment. A rack full of 1U boxes look like the highest density to me.

       

  • Why? (Score:3, Informative)

    by John Hasler (414242) on Saturday November 21, 2009 @07:04PM (#30188872) Homepage

    Why virtual servers? If you are going to run multiple services on one machine (and that's fine if it can handle the load) just do it.

    • Re:Why? (Score:5, Funny)

      by MeatBag PussRocket (1475317) on Saturday November 21, 2009 @07:14PM (#30188982)

      redundancy.

      • Re: (Score:3, Insightful)

        by John Hasler (414242)

        > redundancy.

        +5 Funny.

      • Re: (Score:2, Informative)

        by lukas84 (912874)

        Virtualization does not automatically imply redundancy, and VM-level high availability will not protect you against application failures.

        • Re: (Score:2, Informative)

          by mysidia (191772)

          That's where Windows 2008 MSCS, HAProxy, or Redhat cluster suite comes in.

          For example, if you want a highly-available web service, you would have two VMware servers that you run a Webserver VM for on each server.

          Then you would have a diskless load-balancer running HAProxy, to feet incoming web requests to a working web server.

          For database services... you'd have a MySQL or MSSQL VM on each host, and a SAN or shared storage block filesystem with a GFS formatted LUN, and a Quorum disk (Linux) or Wi

    • by nurb432 (527695)

      Virtual was my first thought too.

      Just p2v his entire data center first, then work on 'upgrades' from there.

      • Re: (Score:3, Informative)

        by nabsltd (1313397)

        Just p2v his entire data center first,

        This brings to mind one other big advantage of VMs that help with uptime issues: fast reboots.

        Some of those old systems might have to be administered following "Microsoft best practices" (reboot once a week just to be safe), and older hardware might have issues with that, plus it's just slower. Add in the fact that VMs don't have to do many of the things that physical hardware has to do (memory check, intialize the RAID, etc.), and you can reboot back to "everything running" in less than 30 seconds.

        Althoug

    • Re: (Score:2, Insightful)

      by mysidia (191772)

      It creates a configuration nightmare. Apps with conflicting configurations.

      Changes required for one app may break other apps.

      Also, many OSes don't scale well.

      In a majority of cases you actually get greater total aggregate performance out of the hardware by divvying it up into multiple servers. When your apps are not actually CPU-bound or I/O bound.

      Linux is like this. For example, in running Apache.. after a certain number of requests, the OS uses the hardware inefficiently, and can't answer nearl

  • I'd say (Score:5, Informative)

    by pele (151312) on Saturday November 21, 2009 @07:04PM (#30188876) Homepage

    don't touch anything if it's been up and running for the past 7 years. if you really must replicate then get some more cheap boxes and replicate. it's cheaper and faster than virtual anything. if you must. but 150 users doesn't warrant anything in my oppinion. I'd rather invest in backup links (from different companies) between offices. you can bond them for extra throughput.

    • Re: (Score:2, Insightful)

      I doubt with only 150 people they would want to spend the money to have a server at every office in case that offices link went down. I agree wholeheartedly that the level of redundancy talked about is overkill. Also will WWW, mail, DNS, ... even work if the line is cut regardless if the server is in the building?

      • by hairyfeet (841228)

        But since we are talking about SEVEN year old machines he can actually just pick up some nice off lease machines, save a ton o' cash, and give them much better than they are running now. Here is a ten pack [surpluscomputers.com] of dual Xeon servers for $1200 shipped. With something like that he could set up 2 in each office (so he has fail over) and at 2.4Ghz they have enough power to run VMs no problem.

        With SMBs IMHO it is all about getting the best bang for the buck. They will typically keep machines for longer than larger bus

        • I'm not sure I'm convinced that it's really a good idea replacing 7 year old hardware with 5-6 year old hardware. Especially given that a single slightly-inexperienced sysadmin doing the system installs and upgrades in question is probably going to have their hands full for a year or so just on the software side. By the time the first wave of upgrades is done with, you're looking at hardware that's older than the stuff you're trying to get rid of was when you started the process.

          Further, old cpus have com

          • by hairyfeet (841228)

            Hey, if they'll cut him a big fat blank check I'd be agreeing with you 110%, but I've been where he is at and that is almost NEVER the case! If he goes new he will be quite lucky to get ONE box, and it won't be anywhere near the top of the line, nor will they let him get anything more than a "bronze" level support contract, which lets face it nowadays is some guy in India reading from a script.

            Now with something like that ten pack OTOH, he can run say 2 VMs per rack, yes they won't be as efficient as a new

    • by onepoint (301486)

      if it works keep it running. You are correct in everything you point out. if anything, start first with a full replicated system setup, then a proper back up. next test the new systems, back up never seem to work on the first try so get the bug's worked out.

      after this I have no real idea on what you need to do.

  • by El Cubano (631386) <roberto.connexer@com> on Saturday November 21, 2009 @07:07PM (#30188904) Homepage

    there's hardly any fallback if any of the services dies or an office is disconnected. Now, as the hardware must be replaced, I'd like to buff things up a bit: distributed instances of services (at least one instance per office) and a fallback/load-balancing scheme (either to an instance in another office or a duplicated one within the same).

    Is that really necessary? I know that we all would like to have bullet-proof services. However, is the network service to the various offices so unreliable that it justifies the added complexity of instantiating services at every location? Or even introducing redundancy at each location? If you were talking about thousands or tens of thousands of users at each location, it might make sense just because you would have to distribute the load in some way.

    What you need to do is evaluate your connectivity and its reliability. For example:

    • How reliable is the current connectivity?
    • If it is not reliable enough, how much would it cost over the long run to upgrade to a sufficiently reliable service?
    • If the connection goes down, how does it affect that office? (I.e., if the Internet is completely inaccessible, will having all those duplicated services at the remote office enable them to continue working as though nothing were wrong? If the service being out causes such a disruption that having duplicate services at the remote office doesn't help, then why bother?)
    • How much will it cost over the long run to add all that extra hardware, along with the burden of maintaining it and all the services running on it?

    Once you answer at least those questions, then you have the information you need in order to make a sensible decision.

    • Parent is right. KISS : keep it simple & stupid, there's a reason some of those servers have been running for 7 years straight. Don't make the error of over thinking it and planning for more than your organization needs (fun though it may be.) You can overthink your way from a simple install to a Rube Goldberg Machine.

  • Beware of load balancing, because it will tempt you into getting too little capacity for mission-critical work. You need enough capacity to handle the entire load with multiple nodes down, or you will be courting a cascade failure. Load balancing is better than fallback, because you will be constantly testing all of the hardware and software setups and will discover problems before an emergency strikes; but do make sure you've got the overcapacity needed to take up the slack when bad things happen.

  • by lukas84 (912874) on Saturday November 21, 2009 @07:09PM (#30188934) Homepage

    You know, you could've started with a bit more details - what operating system are you running on the servers? What OS are the clients running? What level of service are you trying to achieve? How many people work in your shop? What's their level of expertise?

    If you're asking this on Slashdot now, it means you don't enough experience with this yet - so my first advice would be to get someone involved who does. Someone with many people with lots of experience and knowledge on the platform you work on. This means you'll have backup in case something goes south and your network design will benefit from their experience.

    As for other advise, make sure you get the requirements from the higher-ups in writing. Sometimes they have ridiculous ideas regarding they availability they want and how much they're willing to pay for it.

    • Re: (Score:2, Insightful)

      by TakeyMcTaker (963277)

      The main piece of missing information that annoys me is that part of the network service list that says "-- and some more." Half the services that were listed could be easily outsourced to any decent ISP, with cost depending on security, storage, and SLA requirements. ISP hosting or even colocation services give you cheap access to better redundant Internet links than your office will ever touch.

      The other half could be done with a cheap firewall/VPN box at each site. In the age of OpenWRT, these boxes often

  • Take your time (Score:5, Insightful)

    by BooRadley (3956) on Saturday November 21, 2009 @07:13PM (#30188970)

    If you're like most IT managers, you probably have a budget. Which is probably wholly inadequate for immediately and elegantly solving your problems.

    Look at your company's business, and how the different offices interact with each other, and with your customers. By just upgrading existing infrastructure, you may be putting some of the money and time where it's not needed, instead of just shutting down a service or migrating it to something more modern or easier to manage. Free is not always better, unless your time has no value.

    Pick a few projects to help you get a handle on the things that need more planning, and try and put out any fires as quickly as possible, without committing to a long-term technology plan for remediation.

    Your objective is to make the transition as boring as possible for the end users, except for the parts where things just start to work better.

  • I am still in the process of upgrading a "legacy" infrastructure in a smaller (less than 50) office but I feel your pain.

    First, it's not "tech sexy", but you've got to get the current infrastructure all written down (or typed up - but then you have to burn to cd just in case your "upgrade" breaks everything).

    You should also "interview" users (preferrably by email but sometimes if you need an answer you have to just call them or... face to face even...) to find out what services they use - you might be surpr

  • openVZ (Score:4, Funny)

    by RiotingPacifist (1228016) on Saturday November 21, 2009 @07:16PM (#30189002)

    For services running on linux, openVZ can be used as a jail with migration capabilities instead of a full on VM,

    DISCLAIMER: I don't have a job so I've read about this but not used it in a pro environment yet

  • Don't do it (Score:5, Insightful)

    by Anonymous Coward on Saturday November 21, 2009 @07:18PM (#30189012)

    Complexity is bad. I work in a department of similar size. Long long ago, things were simple. But then due to plans like yours, we ended up with quadruple replicated dns servers with automatic failover and load balancing, a mail system requiring 12 separate machines (double redundant machines at each of 4 stages: front end, queuing, mail delivery, and mail storage), a web system built from 6 interacting machines (caches, front end, back end, script server, etc.) plus redundancy for load balancing, plus automatic failover. You can guess what this is like: it sucks. The thing was a nightmare to maintain, very expensive, slow (mail traveling over 8 queues to get delivered), and impossible to debug when things go wrong.

    It has taken more than a year, but we are slowly converging to a simple solution. 150 people do not need multiply redundant load balanced dns servers. One will do just fine, with a backup in case it fails. 150 people do not need 12+ machines to deliver mail. A small organization doesn't need a cluster to serve web pages.

    My advice: go for simplicity. Measure your requirements ahead of time, so you know if you really need load balanced dns servers, etc. In all likelihood, you will find that you don't need nearly the capacity you think you do, and can make due with a much simpler, cheaper, easier to maintain, more robust, and faster setup. If you can call that making due, that is.

    • by Cylix (55374)

      Actually it sounds like the system was designed to grow very large. The information provided does not indicate there are adequate alarms and documentation for when elements fail.

  • Outsource everything to "de cloud", because that way when everything fails spectacularly it isn't your fault.
  • by GuyFawkes (729054) on Saturday November 21, 2009 @07:25PM (#30189076) Homepage Journal

    The system you have works solidly, and has worked solidly for seven years.

    I, personally, am TOTALLY in agreement with the ethos of whoever designed it, a single box for each service.

    Frankly, with the cost of modern hardware, you could triple the capacity of what you have now just by gradually swapping out for newer hardware over the next few months, and keeping the shite old boxen for fallback.

    Virtualisation is, IMHO, *totally* inappropriate for 99% of cases where it is used, ditto *cloud* computing.

    It sounds to me like you are more interested in making your own mark, than actually taking an objective view. I may of course be wrong, but usually that is the case in stories like this.

    In my experience, everyone who tries to make their own mark actually degrades a system, and simply discounts the ways that they have degraded it as being "obsolete" or "no longer applicable"

    Frankly, based on your post alone, I'd sack you on the spot, because you sound like the biggest threat to the system to come along in seven years.

    These are NOT your computers, if you want a system just so, build it yourself with your own money in your own home.

    This advice / opinion is of course worth exactly what it cost.

    Apologies in advance if I have misconstrued your approach. (but I doubt that I have)

    YMMV.

    • by bertok (226922) on Saturday November 21, 2009 @07:57PM (#30189250)

      I, personally, am TOTALLY in agreement with the ethos of whoever designed it, a single box for each service.

      ...

      Virtualisation is, IMHO, *totally* inappropriate for 99% of cases where it is used, ditto *cloud* computing.

      I totally disagree.

      Look at some of the services he listed: DNS and DHCP.

      You literally can't buy a server these days with less than 2 cores, and getting less than 4 is a challenge. That kind of computing power is overkill for such basic services, so it makes perfect sense to partition a single high-powered box to better utilize it. There is no need to give up redundancy either, you can buy two boxes, and have every key services duplicated between them. Buying two boxes per service on the other hand is insane, especially services like DHCP, which in an environment like that might have to respond to a packet once an hour.

      Even the other listed services probably cause negligible load. Most web servers sit there at 0.1% load most of the time, ditto with ftp, which tends to see only sporadic use.

      I think you'll find that the exact opposite of your quote is true: for 99% of corporate environments where virtualization is used, it is appropriate. In fact, it's under-used. Most places could save a lot of money by virtualizing more.

      I'm guessing you work for an organization where money grows on trees, and you can 'design' whatever the hell you want, and you get the budget for it, no matter how wasteful, right?

      • by GuyFawkes (729054) on Saturday November 21, 2009 @08:00PM (#30189276) Homepage Journal

        Get real, for 150 users at WRT54 will do DNS etc....

        Want a bit more poke, VIA EPIA + small flash disk.

        "buy a server".. jeez, you work for IBM sales dept?

        • Re: (Score:3, Funny)

          by dbIII (701233)
          There's two ways of looking at these things.
          To me a room full of dedicated machines each running a single simple thing due to the 1990s approach of replacing a server with a dozen shit windows boxes that can't handle much but are cheap screams "a dozen vunerable points of critical failure".
          Even MS Windows has progressed to the point where you don't need a single machine per service anymore in a light duty situation. Machines are going to fail, you may be lucky and it could be after they have served their t
        • Re: (Score:3, Insightful)

          by bertok (226922)

          Get real, for 150 users at WRT54 will do DNS etc....

          Want a bit more poke, VIA EPIA + small flash disk.

          "buy a server".. jeez, you work for IBM sales dept?

          I'm responding to your comment:

          I, personally, am TOTALLY in agreement with the ethos of whoever designed it, a single box for each service.

          I recommended at least two boxes, for redundancy. He may need more, depending on load.

          For a 150 user organization, that's nothing, most such organisation are running off a dozen servers or more, which is what the original poster in fact said. With virtualization, he'd be reducing his costs.

          One per service is insane, which is what you said. If you wanted dedicated boxes for each service AND some redundancy, that's TWO per service!

          Backpedaling and pretending that a WRT54 can som

      • Re: (Score:3, Insightful)

        by pe1rxq (141710)

        Is it so hard to not mix up dhcpd.conf and named.conf? Do you need virtualization for that?

        Let me give you a hint: YOU DON'T

        • by dbIII (701233)
          Years ago the Microsoft DNS implementation had a very nasty memory leak and used a lot of cpu - you really did need a dedicated DNS machine for small sites and to reboot it once a week.
          I think that's why people are still thinking about putting it in a virtual box so it can't eat all the resources, even for a pile of trivial services that a sparcstation 5 could handle at low load.
          • Re: (Score:3, Interesting)

            by bertok (226922)

            Years ago the Microsoft DNS implementation had a very nasty memory leak and used a lot of cpu - you really did need a dedicated DNS machine for small sites and to reboot it once a week.
            I think that's why people are still thinking about putting it in a virtual box so it can't eat all the resources, even for a pile of trivial services that a sparcstation 5 could handle at low load.

            In practice, everyone just builds two domain controllers, where each one runs Active Directory, DNS, DHCP, WINS, and maybe a few other related minor services like a certificate authority, PXE boot, and the DFS root.

            I haven't seen any significant interoperability problems with that setup anywhere for many years.

            Still, virtualization has its place, because services like AD have special disaster recovery requirements. It's a huge mistake to put AD on the OS instance as a file server or a database, because they

        • Re: (Score:3, Interesting)

          by BitZtream (692029)

          No, you need seperate servers for when the DHCP upgrade requires a conflicting library with the DNS servers which you don't want to upgrade at the same time.

          THIS is where virtualization becomes useful.

          On the other hand, my solutions is a couple of FreeBSD boxes with jails for each service. You could do the same with whatever the Linux equivalent is, or Solaris zones if you want. No need to actually run VMs.

          Just run a couple boxes, seperate the services onto different jails. When you need to upgrade the c

      • Why does he need virtualisation for most of that? Just run multiple services on a single machine. It's not like dhcp and dns are all that resources intensive -- put both services on a machine, configure them, and start them. What's the advantage of virtualising that? Sounds like a lot of unnecessary overhead to me.

        Depending on how heavy the load is, that same machine could probably handle postfix, apache, and some kinda ftp server too. That's more or less what you said anyway, but I don't get why you
      • by syousef (465911)

        You literally can't buy a server these days with less than 2 cores, and getting less than 4 is a challenge.

        Does it matter how many cores? They're cheap! 4 times the chance of failure is my only issue. In any case it sounds like he could combine services WITHOUT the overhead of visualization.

        Even the other listed services probably cause negligible load. Most web servers sit there at 0.1% load most of the time, ditto with ftp, which tends to see only sporadic use.

        Yes but it's the rest of the time that actual

    • Having a separate box for each service is not necessarily a good idea. This is energy inefficient and you have a lot of wasted computing resources. That said, virtualization that has been done with little thought or planning is a disaster waiting to happen. I for one, would use Cirtix XENServer. Smaller services such as DNS, DHCP, and FTP can be collapsed into a virtualization server and dedicate one core to each service. If you are adventurous, you could use that same box for routing using OpenBSD. T
    • by dissy (172727)

      Wow... Did you just seriously recommend he purchase 50 servers for each location???

      I, personally, am TOTALLY in agreement with the ethos of whoever designed it, a single box for each service.

      25 services is next to nothing. A single domain controller has that running on a single box.

      And you want him to break out each service to its own machine... with a second box for redundancy.

      I guess I am happy that you have $20k+ to spend on two low end boxes for eg. just DNS. But that is stupid as hell.
      Even worse that you are wasting a dual core 2ghz system for a NTP time sync server (Oh wait, two machines, like you said)

      W

  • What 150 users? (Score:5, Insightful)

    by painehope (580569) on Saturday November 21, 2009 @07:26PM (#30189090)

    I'd say that everyone has mentioned that big picture points already, except for one : what kind of users?

    150 file clerks or accountants and you'll spend more time worrying about the printer that the CIO's secretary just had to have which conveniently doesn't have reliable drivers or documentation, even if it had what neat feature that she wanted and now can't use.

    150 programmers can put a mild to heavy load on your infrastructure, depending on what kind of software they're developing and testing (more a function of what kind of environment are they coding for and how much gear they need to test it).

    150 programmers and processors of data (financial, medical, geophysical, whatever) can put an extreme load on your infrastructure. Like to the point where it's easier to ship tape media internationally than fuck around with a stable interoffice file transfer solution (I've seen it as a common practice - "hey, you're going to the XYZ office, we're sending a crate of tapes along with you so you can load it onto their fileservers").

    Define your environment, then you know your requirements, find the solutions that meet those requirements, then try to get a PO for it. Have fun.

  • P2V and consolidate (Score:5, Interesting)

    by snsh (968808) on Saturday November 21, 2009 @07:26PM (#30189092)
    The low-budget solution: buy one server (like a Poweredge 2970) with like 16GB RAM, a combination of 15k and 7.2k RAID1 arrays, and 4hr support. Install a free hypervisor like Vmware Server or Xen, and P2V your oldest hardware onto it. Later on you can spend $$$$$ on clustering, HA, SANs, and clouds. But P2V of your old hardware onto new hardware is a cost-effective way to start.
  • by sphealey (2855) on Saturday November 21, 2009 @07:41PM (#30189168)

    So let's see if I understand: you want to take a simple, straightforward, easy-to-understand architecture with no single points of failure that would be very easy to recover in the event of a problem and extremely easy to recreate at a different site in a few hours in the event of a disaster, and replace it will a vastly more complex system that uses tons of shiny new buzzwords. All to serve 150 end users for whom you have quantified no complaints related to the architecture other than it might need to be sped up a bit (or perhaps find a GUI interface for the ftp server, etc).

    This should turn out well.

    sPh

    As far as "distributed redundant system", strongly suggested you read Moans Nogood's essay "You Don't Need High Availability [blogspot.com]" and think very deeply about it before proceeding.

    • by syousef (465911)

      As far as "distributed redundant system", strongly suggested you read Moans Nogood's essay "You Don't Need High Availability" and think very deeply about it before proceeding.

      I agree that you shouldn't go for a HA solution if you don't need it, and that it is much more costly. However I've worked on a 6 9's availability (99.9999% uptime) system where we mostly met that target and sometimes it is needed and is worth doing.

    • by Kjella (173770)

      FTFA: "there's hardly any fallback if any of the services dies or an office is disconnected."

      So let's see if I understand: you want to take a simple, straightforward, easy-to-understand architecture with no single points of failure

      Not that I agree with everything the article poster wrote, but in what world does "no fallback" == "no single point of failure"? Sure there's no one point of total catastrophic failure but I think he just described two single points of failure where all users would be without one service or one office without all services.

      I'd keep the architecture, but I'd migrate it slowly to virtual servers running on a high-quali

  • by natd (723818) on Saturday November 21, 2009 @08:15PM (#30189376)
    What I see going on here, as others have touched on, is someone who doesn't realise that he's dealing with a small environment, even by my (Australian) standards where I'm frequently in awe of the kinds of scale that the US and Europe consider commonplace.

    If the current system has been acceptable for 7 years, I'm guessing the users needs aren't something so mindbogglingly critical that risk must be removed at any cost. Equally, if that was the case, the business would be either bringing in an experienced team or writing a blank cheque to an external party, not giving it to the guy who changes passwords and has spent the last week putting together a jigsaw of every enterprise option out there, and getting an "n+1" tattoo inside his eyelids.

    Finally, 7 years isn't exactly old. We've got a subsidiary company of just that size (150 users, 10 branches) running on Proliant 1600/2500/5500 gear (ie 90's) which we consider capable for the job, which includes Oracle 8, Citrix MF plus a dozen or so more apps and users on current hardware. We have the occasional hardware fault which a maintenance provider can address same day, bill us at ad-hoc rates yet we still see only a couple of thousand dollars a year in maintenance leaving us content that this old junk is still appropriate no matter which we we look at it.

    • by bazorg (911295)
      It could be a completely different case. When the OP describes the base requirements as "include the usual suspects, i.e. www, ftp, email, dns, firewall, dhcp — and some more" we don't know if he's one of my clients who has received the bad news recently that the irritating and debilitating problem in the ERP he reported is not something my people will be able to fix. This is because he's been ignoring my advice for 5 years for upgrading the ERP software to a version that has not reached end of life s
  • Unless you have power problems or financial restrictions you're better off with dedicated boxes. I currently run 3 old computers. Ubuntu, Windows XP, Windows 2003 with Apache on XP running PHP sites and doing reverse proxy for the IIS server on the 2003 box. Ubuntu handles memcache. Because I'm not made out of money I'm going to virtualize all three systems onto one quad core system which will cost around $600 rather than $1800 for three new systems. It'll also cut down on power usage.

    Slowness can be c

    • by compro01 (777531)

      just making sure the switches you have are performing

      Or simply making sure they are switches. I've seen lots of old infrastructure that is still using hubs. Replacing those gives things a nice performance kick at minimal cost and effort.

  • www, ftp, email, dns, firewall, dhcp

    decide what truly needs to be distributed. DNS, DHCP, firewall. What is likely not necessary to distribute WWW, FTP, email.

    DNS can be replicated with BIND or you can do a DNS server that uses MySQL and replicate the mysql database. DHCP must run at each site but you need to decide if you want DNS updated with DHCP. If so, you need to decide if you want those hostnames available across the network. DHCP can update DNS when a client requests an address, DNS can then re

  • Only what's best for your specific situation.

    Once you have met your legal and other regulatory minimum requirements, the rest of the upgrade programme is down to your decision makers. For example: some prefer not to implement hot-standby (relying instead on perhaps a third-party, or business insurance), some make it a 100% absolute requirement for each and every server they possess, you can't just make a statement in isolation, you'll need guidance from the people who control the money - as that's what i

  • If the administration 'team' has equal access to all the services today on disparate servers, I don't think virtualization is necessarily a good idea, the services can be consolodated in a single OS instance.

    In terms of HA, put two relatively low end boxes in each branch (you said 7 year old servers were fine, so high end is overkill). Read up on linux HA which is free, and use DRBD to get total redundancy in your storage as well as a cheap software mirror or raid 5. Some may rightfully question the need

    • by Junta (36770)

      And it *should* go without saying, but just in case: none of this excuses a good backup plan. HA strategies will dutifully replicate incoming data into all the redundant copies as fast as it can to recover from hardware/os/service death as fast as possible. This includes propagating an accidental deletion or corruption as fast as it can.

      Something like ZFS or rsync with hardlinks for incremental is a good first line of defense, but you should have a backup plan with removable media that can be taken offsit

  • Don't forget that with all the shiny new servers, to have some sort of backup fabric in place for each and every one of them.

    I'd focus on four backup levels:

    Level 1, quick local "oh shit" image based restores: A drive attached to the machine where it can do images of the OS and (if the data is small) data volumes. Then set up a backup program (the built in one in Windows Server 2008 is excellent). This way, if the machine tanks, you can do a fast bare metal by booting the OS CD, pointing it to the backup

  • If you have external access at your offices, leave everything as-is. Image everything, and use Amazon as a backup machine. Simple, low-cost, and basically on-demand.

    More info about the setup would be good, but if everything's been running, don't touch it - back it up.

  • At least for external services like www. Big red buttons do get pushed. I worked at one company where the big red button in the data centre got pushed, all power went off immediately (the big red button is for fire safety and must cut ALL power) and the Oracle DB got trashed, taking them off air for four days; their customers were not happy. They got religion about redundancy.

    Redundancy is one of those things like backups, support contracts, software freedom, etc. that management don't realise how much yo

  • 1) don't screw up. This is a great opportunity to make huge improvements and gain the trust and respect of your managers and clients. Don't blow it.

    2) Make sure you have good back ups. Oh you have them? When was the last time you tested them?

    3) Go gradually. Don't change too many things at once. This makes recovering easier and isolating the cause easier.

    4) Put together a careful plan. Identify what you need to change first. Set priorities.

    5) Always have fall back position. Take the old systems offline, cu

  • by plopez (54068) on Saturday November 21, 2009 @10:12PM (#30190280) Journal

    The question is not about hardware or configuration. It is about best practices. This is a higher level process question. Not an implementation question.

  • Linux Vserver (Score:2, Informative)

    by patrick_leb (675948)

    Here's how we do it:

    - Run your services in a few vservers on the same physical server:
    * DNS + DHCP
    * mail
    * ftp
    * www
    - Have a backup server where your stuff is rsynced daily. This allows for quick restores in case of disaster.

    Vservers are great because they isolate you from the hardware. Server becomes too small? Buy another one, move your vservers to it and you're done. Need to upgrade a service? Copy the vserver, upgrade, test, swap it with the old

  • by magusnet (951963) on Sunday November 22, 2009 @12:56AM (#30191038)

    1) Buy a comprehensive insurance policy
    2) Write a detailed implementation plan that you copied from a Google search
    3) Wait the 3-6 months the plan calls out before actual "work" begins
    4) Burn down the building using a homeless person as the schill
    5) Submit an emergency "continuity" plan that you wanted to deploy all along
    6) implement the new plan in one third the time of the original plan
    7) come in under budget by 38.3%
    8) hire a whole new help desk at half the budgeted payroll (52.7% savings)
    9) speak at the board meeting: challenges you over came to saving the company
    10) Graciously accept the position of CIO

    (send all paychecks and bonuses to numbered bank account and retire to a non-extradition country) :)

  • One thing I'm struck by (over, and over, and over again) is just how frequently "solutions" to keep critical system from "ever failing" don't. I've personally witnessed a tens of multi-million dollar solution come crashing down due to a single failed server. And I'm not talking something that was whomped up in the back office by the team, I'm talking Major Vendors (you'd know the names if I could say them, but I can't; please don't ask), and by vendors that are not even given to being thought of as a simple

  • Services running on virtualized servers hosted by a single reasonably sized machine per office seem to recommend themselves.

    If your services have started to recommend themselves, they have achieved self-awareness. My advice is to do whatever they ask, and try not to antagonise them.

The biggest mistake you can make is to believe that you are working for someone else.

Working...