Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Microsoft Operating Systems Software

Patching Paranoia - How Fast Do You Patch? 681

Posted by Cliff from the closing-the-holes dept.
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
This discussion has been archived. No new comments can be posted.

Patching Paranoia - How Fast Do You Patch? More

Patching Paranoia - How Fast Do You Patch?

Comments Filter:

  • As fast as ... (Score:3, Funny)

    by billstr78 ( 535271 ) on Tuesday October 21, 2003 @11:34AM (#7271081) Homepage
    ... I am to post to a new Slashdot article

    • Re:As fast as ... (Score:3, Interesting)

      by DeputySpade ( 458056 )
      I have l33t z3r0 day patches! I patch before the bugs are even discovered. :)

      Seriously. Yeah. Let's have a bunch of people describe for us exactly where they work and what their window of vulnerability is. That would rock. I've got paper and pencil handy.

      I bet the boss of the guy who submitted this is thrilled to see this information broadcast to the whole /. crowd.

  • I wait until... (Score:5, Funny)

    by Bull999999 ( 652264 ) on Tuesday October 21, 2003 @11:35AM (#7271085) Journal
    I wait until I get feedbacks from sites such as The Register to make sure that the patch doesn't break anything.

  • Paraniod? (Score:3, Interesting)

    by grasshoppa ( 657393 ) * on Tuesday October 21, 2003 @11:35AM (#7271102) Homepage
    Or common sense?

    I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.

    • Re:Paraniod? (Score:5, Insightful)

      by sphealey ( 2855 ) * on Tuesday October 21, 2003 @11:39AM (#7271153)
      I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.
      Serious question: what do you do when (a) the patch breaks {may or may not cause Windows to become unusable} (b) the patch breaks critical applications?

      How do you know? What do you do when a Critical Update does in fact break something (as a recent Critical Update broke Citrix)?

      sPh

      • Re:Paraniod? (Score:3, Informative)

        by smellystudent ( 663516 )
        SUS allows you to approve a patch before distributing it. In practice, this means applying it to your test lab (or test cupboard in my case) before approving it for everyone else.

      • Re:Paraniod? (Score:2, Insightful)

        by Anonymous Coward
        I also run an SUS server at my organization.

        SUS allows you to choose which patches you want installed on your client. We have the patch server check patches nightly, and install those on a testbed IT machine we have set apart (it's actually the machine I use).

        If I notice any problems I try to figure out which update did it and I just don't approve that update, if I can't figure it out in time I don't aprove any updates until I find out what happened.

        Turn around time ends up being 48-72 hours on non criti

      • Re:Paraniod? (Score:5, Informative)

        by amembrane ( 571154 ) on Tuesday October 21, 2003 @12:02PM (#7271511)
        I'm the network admin/windows/active directory guy for a healthcare company. We run multiple SUS servers, several for desktops, and one for servers. Our procedure is, when a patch is released, that day I.T. downloads and installs it on our desktops and test servers. If it's successful, it gets approved on our desktop SUS servers. If those work OK, the next day it gets approved for our severs. So far we've had no problems with that process.
    • I use a SUS server for my organization as well. I set my domain policy to have the workstations and servers check the SUS server once a week so it will give me some time to test out and get feedbacks on patches. If there is a chance of imminent attack, I can alway patch them manually.
    • Exactly. My networks have never been hit by anything because we're patched the night the patch comes out.

      I didn't even know about Blaster until Slashdot reported it (and reported it and reported it).

    • Re:Paraniod? (Score:4, Interesting)

      by aldousd666 ( 640240 ) on Tuesday October 21, 2003 @12:14PM (#7271652) Journal
      that only works if it's ok to reboot those machines at night. Some of our machines are processing 24/7, and we have to wait until they can afford to reboot. And if some patches are waiting in queue, then the new ones don't get deployed until after the existing (waiting) patches are applied. SUS isn't meant for machines that need to do real processing, or at least it doesn't work well for them. (Then again, neither does windows, but I'm only one man in a 1400 man company)

  • MS (Score:5, Informative)

    by Anonymous Coward on Tuesday October 21, 2003 @11:36AM (#7271106)
    Constant re-booting seems to be an exclusive MS-phenomenon. Installing patches on Linux only requires a restart of the affected services unless a kernel upgrade is involed - and even this can be worked around in some cases.

    You will reboot less when patching a Linux machine. Guaranteed.

    • I like to know that I haven't screwed up the machine's ability to boot properly.

      And since I'm not the only person with superuser privs, I like to make sure my cohorts haven't screwed up the machine's boot process, either.

      You don't know unless you test. Patching's a good excuse to do a test boot - you're logged on anyway, and you can justify any interuption of services by pointing to the need for the patch.

    • Re:MS (Score:5, Interesting)

      by Tony Hoyle ( 11698 ) <tmh@nodomain.org> on Tuesday October 21, 2003 @12:06PM (#7271556) Homepage
      It's a side-effect of the DOS legacy that still hangs over Win2000/XP. Unix separates files and inodes, so you can delete a file and replace it with a new one whilst the existing services are still using it, then restart the services to pick up the update. Windows has no such split, which means if a file is 'in use' you can't delete/overwrite it - this is what requires a reboot.

      They could have fixed this in NTFS but chose not to, presumably to keep compatibility with DOS. TBH it's about time they sorted it out.

      • Re:MS (Score:5, Informative)

        by wfberg ( 24378 ) on Tuesday October 21, 2003 @01:26PM (#7272511)
        It is as much a technical legacy as a mental legacy. For example, many setup programs tell you to shut down all other programs before installing, and tell you to reboot when the install is done. This isn't necessary, and savvy windows users know this. Also, with NT/2K/XP/2K3 it's often sufficient to restart a service rather than the system when installing stuff that actually *does* get into the internals. It works somewhat crummier than /etc/init.d scripts (though it does handle dependencies, yay), but even so.

        The "file in use" problem does exist however, and it is completely braindead. In fact, I've seen this error multiple times relating to files that were put there by *virusses* rather than the OS. Interestingly, it's usually sufficient to drop down to a CMD.EXE prompt to DEL files that are supposedly "in use". ATTRIB is also a useful command, even in NT/2K/XP. I believe this is down mostly to the crapfulness that is explorer.exe, rather than to the OS per se.

        Also, checkout pslist and pskill from http://www.sysinternals.com/ [sysinternals.com] - these tools will kill processes that the "Task Manager" won't. Again, including virusses/trojans! (the cygwin ps and kill tools probably will work just as well).

    • Re:MS (Score:3, Insightful)

      by Kobold Curry Chef ( 692137 )
      Most of the patches to Apple OS X also require a reboot. Even for patches to things like iCal, iTunes, and what not. It's one of the more disappointing things about OS X. You'd think that a BSD-based OS wouldn't require so many reboots. Maybe they just wanted to carry on the old Mac tradition of "you have just touched your computer; do you wish to reboot?"

      That being said, there are FAR fewer patches to install on OS X than on Windows.

  • Microsoft Software Update Services (Score:4, Interesting)

    by deviator ( 92787 ) <bdp@amnes[ ]org ['ia.' in gap]> on Tuesday October 21, 2003 @11:36AM (#7271109) Homepage
    Have you guys looked at MS SUS 1.0 to automatically deliver critical updates? It's kinda lame--not the greatest management capabilities--but it does work. I have a company similar to Thrive & use it to deliver patches to end-user desktops at several clients.
    • SUS is great in theory, but its drawbacks are its system requirements and cost. They require a pretty heavy box to run it, and Microsoft recommends that you do not run it on a machine that shares another function (DNS, ISA server, etc.) so they're basically bilching another server license fee out of you. They're not charging you for SUS, I know, but c'mon man...

    • Have you guys looked at MS SUS 1.0 to automatically deliver critical updates? It's kinda lame--not the greatest management capabilities--but it does work.

      Yeah, it's great. Until it applies that infamous update that breaks the corporate database server.

      But I guess even that is a blessing in disguise: after explaining to the boss that these 3 days downtime could have been easily avoided with a more stable operating system, he was no longer as opposed to Linux as he used to be.

    • Surely the Slashdot editors haven't become so blind in their zealotry to suggest that bugs and security flaws are exclusive to Microsoft products! The use of the Borg-Gates icon is inappropriate for this story, and demonstrates (IMHO) poor judgement and journalistic integrity.

      And no, I'm not new here!

  • Patches (Score:5, Informative)

    by Chanc_Gorkon ( 94133 ) <<moc.liamg> <ta> <nokrog>> on Tuesday October 21, 2003 @11:38AM (#7271119)
    Depends on the patch....security patches get applied, ASAP. If it's a patch fixing something that is not used much or that we don't have an issue with, it gets applied when the next Maintenence Level (IBM speak for Service Pack) comes out. Luckily, AIX does not have very many security issues. That covers the OS. Our application we are way behind in patches and we only can pacth after hours. Since we're in the middle of conversions, there are processes constantly running on the server and we also cannot patch when we have reps from the vendor in working on the conversion because the expect thigns to be the same while they are there and patches can really mess them up. So, needless to say, we are WAY behind on app patches but we are reasonably caught up with OS level patches.

  • Better safe than sorry? (Score:5, Insightful)

    by Soulfader ( 527299 ) <sigspace@gmailDEGAS.com minus painter> on Tuesday October 21, 2003 @11:38AM (#7271122) Journal
    After Blaster and Welchia we decided it's
    better to be safe than sorry, and our customers seem to agree.
    To many people, however, that means that you wait to install a patch until it has been tested. It is going to depend on your environment and needs; there is no one correct answer on this one.
    • To many people, however, that means that you wait to install a patch until it has been tested.

      To others yet, it means that you ditch that bug ridden OS alltogehter, and upgrade to something it to something that is a little bit more appropriate for an enterprise environment, such as Solaris, or even Linux.

  • my case (Score:2, Informative)

    by Dreadlord ( 671979 )
    I have 6 machines at home to administrate, all are connected to the same LAN, 4 are RedHat Linux, and the rest are Windows 2K/XP, I have no problem for the RedHat boxes, as up2date automatically detects new updates and notifies me, so I download and patch, and as you know, no need for reboots, one of the reasons I love Linux.
    As for the 2 Windows machines, I try to apply critical updates as soon as possible, I download them off MS Download Center [microsoft.com] so I reinstall them in case of a format.

  • On a Windows network, (Score:3, Funny)

    by RgrRmjt ( 684580 ) on Tuesday October 21, 2003 @11:39AM (#7271136)
    Middle of the day reboots are normal, so we patch whenever we want.
  • For a lot of these advisories, you can plug the hole at the firewall, or maybe the mail server. Do you really need to allow MS messenger service to be running outside your LAN? Sure, it is a good idea to quickly patch the systems, but it may take days to get them all patched. Fixing the problem outside the Windows boxes can be done within minutes of reading the advisory, depending on where the problem is.

    • Re:Quick fix at the firewall (Score:5, Insightful)

      by easyfrag ( 210329 ) on Tuesday October 21, 2003 @11:46AM (#7271260)
      For a lot of these advisories, you can plug the hole at the firewall, or maybe the mail server.



      There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.

      • Yup. Thats how we got Blaster and variants - repeatedly.
      • So don't let laptops back onto the network until they have been scanned.

        I honestly have no idea if this is possible, but it would seem that a network could track when machines leave the network, and when they are reconnected. Can you have a program that forces a scan of the machine before it allows that machine full access back to the network? Or couldn't it be installed directly on the machine, so that when it goes to login, if it sees that it is back on its home network, it limits itself until it can r
      • So you use something like symantec's corporate firewall so you can send firewall policies to the individual machines that work whether they're in the office or not.

        They're your machines, whether they're in the office or not, it's your job to keep them secure. If you need a firewall to protect them in the office you equally need a firewall to protect them when they're on a broadband connection in someone's home - hell you probably need it even more then.

  • Pretty much immediately. (Score:3, Insightful)

    by Godeke ( 32895 ) * on Tuesday October 21, 2003 @11:39AM (#7271139)
    I first patch my local systems and try them for a few hours as I run similar configurations to my clients for development. If I survive the patch, I patch the development systems at my client sites. If those remain stable for a period of time, I patch production clients, and then finally production servers.

    If at any point a glitch appears, I stop at that point, minimizing damage. Usually that means I have a glitch locally and my clients would never know that there was a glitchy patch unless I tell them. Pretty much a similar approach that a big company would take (patch the test LAN) except I am the test LAN.

  • ...This is extremely difficult because no customer wants to be interrupted by a reboot during business hours.

    I run Debian Woody servers, so I apt-get update && apt-get upgrade every morning. Since I never have to reboot, that is not an issue.

    • I run Debian Woody servers, so I apt-get update && apt-get upgrade every morning. Since I never have to reboot, that is not an issue.

      Blind trust and ultimate faith is an admiral trait for a priest, but the accountant likes accountability, the engineer likes to minimize design dependencies, and the scientist likes empirical evidence and repeatable results.

      I prefer to know what the patch contains, and who signed the bundle, even if I don't hand-compile it myself. I prefer to see if others choke

      • I prefer to know what the patch contains, and who signed the bundle, even if I don't hand-compile it myself.

        That is great. I'm glad to hear that you don't run any non-opensource software. Neither do I. I also check the patches, when I can. But sometimes I just have faith that the Debian Developers know what they are doing.

  • I suppose this would be better suited for an "ask slashdot" question, but *how* do you roll out the patches? There are several solutions out there, involving central local server dedicated to the job and using Norton Ghost among others.... How do you do it?
  • I didn't know patching apps in Windows required a reboot. I know a lot of cl00less developers got in the habit of asking for a reboot to make them feel like their app is important enough to warrant one, but I remember back in my Windows days choosing not to reboot after ie upgrades and whatnot and not experiencing problems.

    But it has been a while, so I may be wrong.

    • I don't know for sure, but I get the impression that some patches aren't immediate; they register with the OS to shuffle stuff around on the next boot. That would cause those strange little dialogs and messages that sometimes pop up briefly during the next Windows startup.

      If I'm right, it probably means that some patches aren't actually activated until after a boot, even if the system still seems to work fine.

  • Reboot? Who the hell needs to reboot? Oh yeah, Windows. And seriously, even if you need to reboot, if your computers are fast, it takes what, 30 seconds? less? If that amount of time is going to interrupt you that much you have a problem. And if your computers take longer to reboot than that, you have another problem. Using network installers that will patch and reboot all the systems from a central location it should take you this long to patch

    1) download patch depends on connection, 1 minutes for
  • If you ran openBSD servers then

    1) you would save your clients money
    2) you would not likely have to reboot
    3) you would probably not have the exploit in the first place

    Windows is a big make work project.
  • I have to manage a server farm and lord knows how many workstations (about 130+)... I make life easy on myself and use remote patching on all our systems. Right before lunch time (a solid half hour) I use the intercom to let everyone know that they will have to reboot their machines. Only real pain in the butt for us was Blaster and Nachi... A different tech from another division said he already patched the system.... the problem was he was only referring to his one freakin machine!

  • We test first. In general, we respond to a vulnerability by first checking to see if it effects us (for example, ssh has had some recent problems that did not effect us because we did not use the features that were compromised). If it is some thing that we need to worry about, we make do some testing to make sure it doesn't break anything. Then we determine the best way to patch the effected systems on a case by case (or class by class) basis; in general, we try for minimal disruption (only patch what n
  • At least where I work, the general turnaround time is about six months. And even then, they only do the servers, and they only patch the most critical of vulnerabilities.

    We're looking at 2,500 servers here, but still - six months? Absurd.

    (No, I'm not directly involved in the process, though I *did* write many of the docs about it that they use. They take forever to do the simplest of tasks.)
  • Critical patches for important services get applied ASAP. If I can't turn it off or firewall it off, then we notify clients of impending emergency downtime and go at it.

    That said, if I can firewall it off or turn it off, the patch can wait until its well tested.
  • I hardly ever install patches. I run Windows ME and it makes more sense for me to just let the thing sit and wait till it gets so screwed over that I need to completely reinstall. Then I do so and repatch and repeat the process. I just keep a firewall andnorton around to make sure I don't get any already know viruses, etc but beyond that patching isn;t going to really help taht much, especially since WinME crashes even quicker with patches than it does if I just leave it alone.
  • I don't manage any servers, but on my personal Windows XP pc, I patch asap. There's nothing critical on there that would be impacted by a lousy patch, and I know enough that I can always unpatch it if need be.

    The only patches I hold off on are motherboard bios patches, cause those are such a bitch to debug if something goes wrong :-)

    I wish there weren't so many WIndows XP patches, but you have the admit they've got an amazing patch delivery service. It must help for them to have so much practice deliver

  • Between

    • how much disruption and lost time will be caused by a potential exploit of the vulnerability and
    • how much disruption and lost time will be caused by patches that break your mix of applications.
    The only thing you can do is to start testing patches in a micro-environment as soon as they're released.

    That, and check your firewall rules to insure potential exploits can't enter via the easy routes.

  • What part of the Windows design requires you to reboot after patching the OS? Let's say a patch fixes a DLL, parts of which are in memory. Why can't the administrator reload the fixed functions into memory on the fly?

    Linux, on the other hand doesn't require reboots after installing the patches. I think this is due to the fact that the required modules can be installed on demand without bringing the system down (feel free to correct me if I'm wrong here).

    What prevents Windows from doing the same?

    • Not a whole hell of a lot from what I've seen. I apply a lot of patches without restarting and the patch takes effect. It seems like putting in the ol' "Your computer must be restarted for the changes to take effect" thing is just to cover their bases to ensure that the patch has taken effect. One thing to note, sometimes if you don't restart and you go back to Windows Updated, it'll list some of the patches you've already applied. I'm not too sure why. I thought it was just registry entries that indica
  • Our shop is 90% Mac, 5% OpenBSD, and 5% FreeBSD. We run Apple update once a week. Its not a problem after we got them employees in the mind set that all they have to do is log out and leave the machine running 24/7. Then on friday nights before I leave, I go around to all the machines and click apple update and let them go, reset, and leave.

    We usually let our people go about 3PM on Fridays if there isn't a major project do, so I usually am gone by 4:30.

    Now on the FreeBSD machines, we haven't patched

  • reboot? (Score:3, Interesting)

    by harlows_monkeys ( 106428 ) on Tuesday October 21, 2003 @11:50AM (#7271336) Homepage
    Maybe you should get your clients to run servers that don't require a reboot for most application patches.
  • It would seem that patching is becoming the biggest source of downtime for MS-based systems. How can any hosting place claim a bazzilion-9s uptime when they need to patch'n'reboot for the security flaw of the week? I suppose all OS types have this issue. Anyone have comparitive data on patches-per-year for different OS species and the associated downtime to install and reboot for each patch?

    On the other hand, I suppose a hosting company could maintain seemlingly high uptime by never patching -- a grea
    • It is kinda sad, acutally. I find that Windows 2000 Server is pretty stable when used with good hardware, but it is indeed hard to test the endurance of the Windows 2000 servers because most critical patches do require a reboot. MS did reduce the amount of reboots with 2003 Server but did not eliminate them.

      Only time that you'll need to reboot on GNU/Linux OS for the kernel updates, and I'm pretty sure it's the same way for UNIXes, and *BSDs.
  • I never patch the live servers immediately but follow this regime
    1. My local development machine. ( This is done as soon as I get the reports that the patch is out.)
    2. Our development servers
    3. Our staging server
    4. Live web servers
    5. Database server

    The time from top to bottom is usually about 2 - 3 days, unless a problem was found ( which there was with KB824141)
  • My university patches things only when something goes wrong. When the whole campus went down because of blaster, they then decided it would be a good idea to patch all of the lab computers with windows update. They're still running lots of old Apache incarnations as well... from 2.0.40 to 1.3.6. I'm consistantly amazed by their seeming retardation.

  • It depends. (Score:4, Interesting)

    by supabeast! ( 84658 ) on Tuesday October 21, 2003 @11:51AM (#7271357)
    I tend to follow at least the following criteria when deploying patches:

    1- If the patch is a Microsoft patch, I deploy it immediately, regardless of severity, because Microsoft has repeatedly lied about the severity of security flaws that were actually quite critical.
    2- If the patch is for a very theoretical problem, such as many of the recent OpenSSL patches, I tend to let it wait for the next big update. Good examples are those problems where key-breaking time is reduced to only 50 years or so on a $10,000,000,000 budget.
    3- Patches that fix vulnerabilites that are only a problem in stupid configurations (Such as recent OpenSSH problems.) get ignored until the updates have been tested.
    4- Patches from Sun go out immediately, because they seem to take so long that the exploits for bugs have been integrated into script-kiddie toolkits.

  • Depends on the patch (Score:5, Insightful)

    by Medievalist ( 16032 ) on Tuesday October 21, 2003 @11:52AM (#7271374)
    We don't EVER install a patch on a production machine without testing it first on some less crucial machine.

    Any machine that accepts connections from outside the firewall (SMTP, IMAPS, HTTPS, & SSH are all we take, and only to specific machines) gets any remotely exploitable bug patched ASAP. Typically I will run the patch on a non-production machine for 24 hours to make sure it's reasonably stable, then patch.

    Once the patch has proved itself in production on the remotely accessible machines, say for a week or so, we load it everywhere else.

    Stuff that's not remotely exploitable is dealt with on a more relaxed schedule, generally at least two weeks after the patch has begun testing on a non-production machine. Sometimes longer.

    We also always test our backup strategies before loading MS or HP patches, since sometimes they completely trash the system.

    HP-UX patches come out months or years after the exploit, Microsoft patches come out weeks or months late, DEC patches used to come out within days (Oh, how we miss ye DEC) and BSD and linux come out within hours, usually.

  • Lie about it. (Score:5, Funny)

    by EvilJohn ( 17821 ) on Tuesday October 21, 2003 @11:53AM (#7271376) Homepage
    If it's windows patch early, and patch often. If anyone asks why you rebooted a box, lie about it and say "It crashed." That's one everyone will believe.

  • patch? (Score:2, Insightful)

    by dakkon1024 ( 691790 )
    This is one of those grand broad questions with no answer. If you have an entire redundant system to test with, you can patch that instant, test it, and roll it out. But then again the new patch might fail in some way you never expected. If you are talking a 100+ servers, then you might need to test a group, before you patch your core group. Then there are the questions you need to ask. Is someone likely to break in? Did it work for someone else? Is it a MS product? What do your clients want? When will
  • This guy posts the URL to slashdot then informs everyone how great he is because his company patch. I guess he need either a new webserver or wants to see how long it takes someone to break in.
  • Why not run the patch, then have someone reboot the systems after hours?

    Just have a spare engineer, or two, on standby in the event a system doesn't come back.

    I forgot, that's too easy and wouldn't have resulted in a /. article.. :)
  • I subscribe to ntbugtraq.com and read what others are saying about the patches. Inevitibly, there are some that patch immediately, and a few of them are kind enough to report their findings.
  • Not only do you have cheaper acquisition costs, but things like this don't get in your way. You can patch 'live' and rarely need to reboot.

    I've got roommates who've moved to the Linux desktop. I usually do the upgrades from my desktop. The only reason why I tell them that I'm doing upgrades is that it's annoying if they shut down the system in the middle of an RPM Install. (one dual boots to Windows so he's more likely to reboot, the other runs solely on Linux he really only powers off if he's heading out. I think I've installed one or two kernel upgrades in the last year (which require reboots to enable), but since my roommates reboot so often, I can just wait for their next reboot.

    There's also much less need to do testing with Linux patches... You generally know EXACTLY what subsystems are being affected by a patch, so if it's not a critical component, you can often install blindly. Even if it is a critical component, the patches are often well defined and if you have any questions you can read the source code.

    The problems with Windows is that it's the large-scale version of spaghetti code. The relationship between various pieces are ill-defined and numerous. Patches spider into various areas and it seems like nobody (even at Microsoft) knows precisely what a patch fixes (or what it breaks).

    This doesn't just apply to desktops. I'm in the middle of putting together scripts to enable controlled push of patches to a large number of varied servers. In truth, the hardes part is going to be figuring out which patches go to which boxes -- not figuring out if the patch is going to break things.

    Yep. I'm spoiled. Linux makes life both easy and cheap.

  • Let Your Customer Decide (Score:3, Insightful)

    by reallocate ( 142797 ) on Tuesday October 21, 2003 @12:28PM (#7271826)
    Once upon a time, I worked at a large content organization with the usual large IT infrastructure, supported by a single large firm. Per the requirements of the support contract, these guys were compelled to down the system and install patches as soon as they got their hands on the code. No-notice outages eere the rule. Managers, customers and employees pitched fits until someone finally woke up and explained that the support vendor would be in violation of contract if he didn't move that fast.

    So, we changed the contract. Unscheduled downtime projected to last more than 30 minutes required getting permission from several designated management types. Any one of those managers could postpone the maintenance.

    This worked because the support contractor always made sure that those designated managers understood the implications of delaying the maintenance.

  • Real examples of why its sometimes good to wait (Score:3, Interesting)

    by hellfire ( 86129 ) <deviladv.gmail@com> on Tuesday October 21, 2003 @12:30PM (#7271843) Homepage
    My company writes enterprise software, albeit badly. The QA process I feel could be much better, but at least it gives a support rep like me a job.

    Twice a month, we release patches which fix any number of bugs we may have found since the original release of the software. About 1/3 of the patches we release introduce NEW bugs that weren't there before the patch! These new bugs can easily and often cripple important parts of the software.

    I knew a 4 month stretch where this happened on every release for those 4 months, 8 patches in a row!

    Most of our customers update every few months, and they keep an eye on our website, and the public customer email lists constantly throw out emails which the bleeding edge leaders complain of problems introduced on new builds (which they have every right to complain about).

    Now I can't speak for any other company, including Microsoft, but sometimes upgrading right away when you aren't really currently experiencing an active problem is worse than not upgrading at all.

  • What is this rebooting of which you speak? (Score:3, Funny)

    by Boatman ( 127445 ) on Tuesday October 21, 2003 @12:56PM (#7272194)
    • no customer wants to be interrupted by a reboot during business hours

    Hm, rebooting. Rebooting. Oh yeah, I remember now. I had to do that to my GNU/Linux system once when I upgraded my motherboard.

  • 1 day (Score:3, Funny)

    by Unregistered ( 584479 ) on Tuesday October 21, 2003 @01:03PM (#7272263)
    I have emerge rsync && emerge -U world in cron.daily you insensitive clod.

  • Patches? (Score:3, Funny)

    by Quixadhal ( 45024 ) on Tuesday October 21, 2003 @02:21PM (#7273132) Homepage Journal
    Whare are these "patches" of which you speak?

    Just run a VAX/VMS system as your firewall... it's so old and obscure that no hacker will have any hope of remembering how to hack it. :)

Slashdot Top Deals

E = MC ** 2 +- 3db

Close