Patching Paranoia - How Fast Do You Patch? 681
selfassembled asks: "I work for an IT group in the Boston area called Thrive Networks. After the most recent exploit was revealed, my company scrambled to get our client's servers patched within 48 hours. This is extremely difficult because no customer wants to be interrupted by a reboot during business hours. Our staff worked after hours to get this patch installed ASAP. How fast do you (or your IT group) install patches for major exploits like this? What do you consider to be an acceptable turn around time for a vulnerability patch that may not even have an exploit yet? After Blaster and Welchia we decided it's better to be safe than sorry, and our customers seem to agree."
As fast as ... (Score:3, Funny)
Re:As fast as ... (Score:3, Interesting)
Seriously. Yeah. Let's have a bunch of people describe for us exactly where they work and what their window of vulnerability is. That would rock. I've got paper and pencil handy.
I bet the boss of the guy who submitted this is thrilled to see this information broadcast to the whole
I wait until... (Score:5, Funny)
Re:I wait until... (Score:2, Informative)
Re:I wait until... (Score:2)
Re:I wait until... (Score:5, Funny)
I think "Breaking Groupwise" is an MS patch all by itself.
"CRITICAL UPDATE: SOME SYSTEMS HAVE GROUPWISE INSTALLED ON THEM. THIS PATCH WILL BREAK GROUPWISE."
Re:I wait until... (Score:4, Funny)
Re:I wait until... (Score:3, Informative)
Re:I wait until... (Score:5, Informative)
BTW, you may want to change your sig because at first, I thought that it was part of the message. Most mods won't know the differents and will mod you flamebait.
Re:I wait until... (Score:5, Funny)
well they should have POSTED about it! jeez!
Re:I wait until... (Score:3, Informative)
Re:I wait until... (Score:3, Interesting)
Let's just say that I approached Service Pack 4 with a great deal of apprehension. I've had good luck with workstation upgrades, but my server experience is decidely mixed.
Re:I wait until... (Score:3, Interesting)
I d
Re:I wait until... (Score:3, Interesting)
He's got an IBM server - probably a big production machine. It's almost certainly a SCSI Raid setup.
It's not possible to plug the array into the regular controller.
In any case, doesn't matter if this would fix it or not. It shouldn't happen EVER.
I'm not sure which is worse, I take the box down to patch, and get heart palpatations when it goes down catestrophically, or someone roots my box.
Either case, I'd be pissed.
Cheers,
Greg
Re:I wait until... (Score:3, Insightful)
Re:I wait until... (Score:3, Informative)
Re:I wait until... (Score:3, Informative)
SP4 broke, among others, the Terminal Services (for win2k TS servers) -- the logins now take over 30 seconds (from 5 sec earlier). During TS sessions the TS freezes few times an hour for around 20-30 seconds at a time, making it unusable for some tasks and wasteful (of time and nerves) for the remaining ones.
Other patches and "upgrades", especially those for IE, have been degrading win98se performance and stability (such a
Re:I wait until... (Score:3, Interesting)
Just goes to show how touch-n-go a windows patch can be..sometimes it borks your system, sometimes it doesn't. There's really no log
Re:I wait until... (Score:2, Informative)
Re:I wait until... (Score:5, Interesting)
We have constant problems with patches where I work because Hpaq/Sun seem to think that the versions of certain software they ship with Solaris/Tru64 are sacrosanct.
Every time we patch our primary DNS server (on an E-250) Sun's patch stomps on our custom build of BIND. Similarly, HPaqs patch kits won't install properly if they involve any patches for sendmail because we got tired of waiting for patches for 8.9.3 (even under 5.1A they stay with 8.9.3!) while we prefer to run our own build of 8.12.10. HPaq is also bad about making security patches depend on their version of the software unnecessarily. As a f'rinstance, I recently installed Aggregate Patch Kit 5 for Tru64 5.1A. It included about a half-dozen patches to fix weaknesses in the init scripts. The patches for the init scripts REFUSED to install until I downgraded sendmail to 8.9.3 configured as it was during the system installation! After the patches were installed, I had to re-upgrade sendmail to our preferred version. To the best of my ability to determine there was absolutely NO reason for those patches to depend on sendmail being at v 8.9.3.
Re:I wait until... (Score:2)
There was a "slow-down-your-system" windows patch [updatexp.com] in April 2003. Took them a while to release a working version of it, too.
Re:I wait until... (Score:2)
Now we have a junk machine that gets the patches first and if it breaks anything we don't roll it out - no matter what it claims to fix (a lot of recent patches haven't actually fixed the bugs they claim to fix... MS don't seem to test them before releasing them).
Mac OS X 10.2.8 (Score:4, Informative)
I reverted to 10.2.6 and all was well once again.
And this was 10.2.8 redeux - remember the first time that it came out, machines were breaking all over the place. (ethernet issues, IDE oddities..)
Re:I wait until... (Score:4, Informative)
One solution I believe is to make every patch and update available separately. In addition provide an update tool with presets that choose only the latest security fixes or feature updates or all updates, and allow administer's to customize their own presets. You are then faced with the issue of dependencies however these can be easily addressed by warnings letting you know what additional software is required and will be installed.
Re:I don't apply these kinds of patches (Score:3, Troll)
I certainly didn't like patching OpenSSH on a machine I can only reach via SSH.
Paraniod? (Score:3, Interesting)
I run a SUS server for my organization, and it checks for patches nightly. The next day, my servers and workstations are patched.
Re:Paraniod? (Score:5, Insightful)
How do you know? What do you do when a Critical Update does in fact break something (as a recent Critical Update broke Citrix)?
sPh
Re:Paraniod? (Score:3, Informative)
Re:Paraniod? (Score:2, Insightful)
SUS allows you to choose which patches you want installed on your client. We have the patch server check patches nightly, and install those on a testbed IT machine we have set apart (it's actually the machine I use).
If I notice any problems I try to figure out which update did it and I just don't approve that update, if I can't figure it out in time I don't aprove any updates until I find out what happened.
Turn around time ends up being 48-72 hours on non criti
Re:Paraniod? (Score:5, Informative)
Re:Paraniod? (Score:2)
The problem is that very few small- and mid-sized organizations have that level of expertise. And the vendors of the vertical apps such orgs tend to use are not very responsive when it comes to
Re:Paraniod? (Score:2)
Re:Paraniod? (Score:3, Interesting)
I didn't even know about Blaster until Slashdot reported it (and reported it and reported it).
Re:Paraniod? (Score:4, Interesting)
MS (Score:5, Informative)
You will reboot less when patching a Linux machine. Guaranteed.
I reboot anyway. I like to reboot. (Score:2)
I like to know that I haven't screwed up the machine's ability to boot properly.
And since I'm not the only person with superuser privs, I like to make sure my cohorts haven't screwed up the machine's boot process, either.
You don't know unless you test. Patching's a good excuse to do a test boot - you're logged on anyway, and you can justify any interuption of services by pointing to the need for the patch.
Re:MS (Score:5, Interesting)
They could have fixed this in NTFS but chose not to, presumably to keep compatibility with DOS. TBH it's about time they sorted it out.
Re:MS (Score:5, Informative)
The "file in use" problem does exist however, and it is completely braindead. In fact, I've seen this error multiple times relating to files that were put there by *virusses* rather than the OS. Interestingly, it's usually sufficient to drop down to a CMD.EXE prompt to DEL files that are supposedly "in use". ATTRIB is also a useful command, even in NT/2K/XP. I believe this is down mostly to the crapfulness that is explorer.exe, rather than to the OS per se.
Also, checkout pslist and pskill from http://www.sysinternals.com/ [sysinternals.com] - these tools will kill processes that the "Task Manager" won't. Again, including virusses/trojans! (the cygwin ps and kill tools probably will work just as well).
Re:MS (Score:3, Informative)
This is just like anonymous temporary files. You open a new t
Re:MS (Score:3, Insightful)
That being said, there are FAR fewer patches to install on OS X than on Windows.
Re:MS (Score:4, Funny)
I just upgraded libc. What do I have to restart?
Microsoft Software Update Services (Score:4, Interesting)
Re:Microsoft Software Update Services (Score:2)
Re:Microsoft Software Update Services (Score:2)
Re:Microsoft Software Update Services (Score:2)
Yeah, it's great. Until it applies that infamous update that breaks the corporate database server.
But I guess even that is a blessing in disguise: after explaining to the boss that these 3 days downtime could have been easily avoided with a more stable operating system, he was no longer as opposed to Linux as he used to be.
Why the 'Microsoft' icon for this story? (Score:2)
And no, I'm not new here!
Patches (Score:5, Informative)
Better safe than sorry? (Score:5, Insightful)
Re:Better safe than sorry? (Score:2)
To others yet, it means that you ditch that bug ridden OS alltogehter, and upgrade to something it to something that is a little bit more appropriate for an enterprise environment, such as Solaris, or even Linux.
Re:Better safe than sorry? (Score:3, Insightful)
In my company, a bad patch can mean 10,000 reps sitting around with nothi
my case (Score:2, Informative)
As for the 2 Windows machines, I try to apply critical updates as soon as possible, I download them off MS Download Center [microsoft.com] so I reinstall them in case of a format.
On a Windows network, (Score:3, Funny)
Quick fix at the firewall (Score:2)
Re:Quick fix at the firewall (Score:5, Insightful)
There's one big gotcha here: notebooks. Your users are firewalled at work but once they get home they're probably wide open. Plug an infected notebook into your network of unpatched machines and a worm will bring you down in seconds.
Re:Quick fix at the firewall (Score:2)
Re:Quick fix at the firewall (Score:2)
I honestly have no idea if this is possible, but it would seem that a network could track when machines leave the network, and when they are reconnected. Can you have a program that forces a scan of the machine before it allows that machine full access back to the network? Or couldn't it be installed directly on the machine, so that when it goes to login, if it sees that it is back on its home network, it limits itself until it can r
Re:Quick fix at the firewall (Score:2)
They're your machines, whether they're in the office or not, it's your job to keep them secure. If you need a firewall to protect them in the office you equally need a firewall to protect them when they're on a broadband connection in someone's home - hell you probably need it even more then.
Re:Quick fix at the firewall (Score:5, Interesting)
The countless whining we get over passwords ("My boss says I dont hafta have one.."), applying updates to desktops(!), removing shit like comet cursor, and the people that toss laptops around and then bitch that they don't have the right laptop after they've broken it.
I'd love to see 2 or 3 people in particular have to sit down in front of the CFO and be told:
1) The computer you broke won't be replaced until you pay for the old one.
2) If you can write a check today, we won't dock your paycheck, but if we do, we'll spread the payment over at least 4 paychecks.
3) Any work you don't get done due to no computer will be considered against you in your next performance review and may be considered grounds for dismissal.
There's lots of reasons not to do it that way, but geeze, if there were real consequences (financially especially) for being a fuckup with computers, I think the users would toe a much tighter line.
Pretty much immediately. (Score:3, Insightful)
If at any point a glitch appears, I stop at that point, minimizing damage. Usually that means I have a glitch locally and my clients would never know that there was a glitchy patch unless I tell them. Pretty much a similar approach that a big company would take (patch the test LAN) except I am the test LAN.
My solution (Score:2)
I run Debian Woody servers, so I apt-get update && apt-get upgrade every morning. Since I never have to reboot, that is not an issue.
Re:My solution (Score:2)
I run Debian Woody servers, so I apt-get update && apt-get upgrade every morning. Since I never have to reboot, that is not an issue.
Blind trust and ultimate faith is an admiral trait for a priest, but the accountant likes accountability, the engineer likes to minimize design dependencies, and the scientist likes empirical evidence and repeatable results.
I prefer to know what the patch contains, and who signed the bundle, even if I don't hand-compile it myself. I prefer to see if others choke
Re:My solution (Score:2)
I prefer to know what the patch contains, and who signed the bundle, even if I don't hand-compile it myself.
That is great. I'm glad to hear that you don't run any non-opensource software. Neither do I. I also check the patches, when I can. But sometimes I just have faith that the Debian Developers know what they are doing.
Answering a question with a question.... (Score:2)
Re:Answering a question with a question.... (Score:5, Informative)
sPh
You have to reboot? (Score:2)
But it has been a while, so I may be wrong.
Re:You have to reboot? (Score:2)
If I'm right, it probably means that some patches aren't actually activated until after a boot, even if the system still seems to work fine.
Reboot? (Score:2)
1) download patch depends on connection, 1 minutes for
If you ran openBSD servers then (Score:2, Offtopic)
1) you would save your clients money
2) you would not likely have to reboot
3) you would probably not have the exploit in the first place
Windows is a big make work project.
Re:If you ran openBSD servers then (Score:3, Funny)
My exp. (Score:2)
We test first (Score:2)
We test first. In general, we respond to a vulnerability by first checking to see if it effects us (for example, ssh has had some recent problems that did not effect us because we did not use the features that were compromised). If it is some thing that we need to worry about, we make do some testing to make sure it doesn't break anything. Then we determine the best way to patch the effected systems on a case by case (or class by class) basis; in general, we try for minimal disruption (only patch what n
At least where *I* work ... (Score:2)
We're looking at 2,500 servers here, but still - six months? Absurd.
(No, I'm not directly involved in the process, though I *did* write many of the docs about it that they use. They take forever to do the simplest of tasks.)
Depends on several factors (Score:2)
That said, if I can firewall it off or turn it off, the patch can wait until its well tested.
My schedule (Score:2)
mypc (Score:2)
The only patches I hold off on are motherboard bios patches, cause those are such a bitch to debug if something goes wrong
I wish there weren't so many WIndows XP patches, but you have the admit they've got an amazing patch delivery service. It must help for them to have so much practice deliver
Subjective Weighing (Score:2)
Between
That, and check your firewall rules to insure potential exploits can't enter via the easy routes.
Why reboot after patching? (Score:2)
Linux, on the other hand doesn't require reboots after installing the patches. I think this is due to the fact that the required modules can be installed on demand without bringing the system down (feel free to correct me if I'm wrong here).
What prevents Windows from doing the same?
Re:Why reboot after patching? (Score:2)
We don't have to patch...that often (Score:2)
We usually let our people go about 3PM on Fridays if there isn't a major project do, so I usually am gone by 4:30.
Now on the FreeBSD machines, we haven't patched
reboot? (Score:3, Interesting)
Impact on downtime statistics (Score:2)
On the other hand, I suppose a hosting company could maintain seemlingly high uptime by never patching -- a grea
Re:Impact on downtime statistics (Score:2)
Only time that you'll need to reboot on GNU/Linux OS for the kernel updates, and I'm pretty sure it's the same way for UNIXes, and *BSDs.
In Stages (Score:2)
1. My local development machine. ( This is done as soon as I get the reports that the patch is out.)
2. Our development servers
3. Our staging server
4. Live web servers
5. Database server
The time from top to bottom is usually about 2 - 3 days, unless a problem was found ( which there was with KB824141)
My University (Score:2)
It depends. (Score:4, Interesting)
1- If the patch is a Microsoft patch, I deploy it immediately, regardless of severity, because Microsoft has repeatedly lied about the severity of security flaws that were actually quite critical.
2- If the patch is for a very theoretical problem, such as many of the recent OpenSSL patches, I tend to let it wait for the next big update. Good examples are those problems where key-breaking time is reduced to only 50 years or so on a $10,000,000,000 budget.
3- Patches that fix vulnerabilites that are only a problem in stupid configurations (Such as recent OpenSSH problems.) get ignored until the updates have been tested.
4- Patches from Sun go out immediately, because they seem to take so long that the exploits for bugs have been integrated into script-kiddie toolkits.
Re:WTF? (Score:3, Informative)
Depends on the patch (Score:5, Insightful)
Any machine that accepts connections from outside the firewall (SMTP, IMAPS, HTTPS, & SSH are all we take, and only to specific machines) gets any remotely exploitable bug patched ASAP. Typically I will run the patch on a non-production machine for 24 hours to make sure it's reasonably stable, then patch.
Once the patch has proved itself in production on the remotely accessible machines, say for a week or so, we load it everywhere else.
Stuff that's not remotely exploitable is dealt with on a more relaxed schedule, generally at least two weeks after the patch has begun testing on a non-production machine. Sometimes longer.
We also always test our backup strategies before loading MS or HP patches, since sometimes they completely trash the system.
HP-UX patches come out months or years after the exploit, Microsoft patches come out weeks or months late, DEC patches used to come out within days (Oh, how we miss ye DEC) and BSD and linux come out within hours, usually.
Lie about it. (Score:5, Funny)
patch? (Score:2, Insightful)
Asking for trouble (Score:2)
Working late into the night? (Score:2)
Just have a spare engineer, or two, on standby in the event a system doesn't come back.
I forgot, that's too easy and wouldn't have resulted in a
subscription (Score:2)
Why Windows has a Higher Cost of Ownership (Score:3, Insightful)
I've got roommates who've moved to the Linux desktop. I usually do the upgrades from my desktop. The only reason why I tell them that I'm doing upgrades is that it's annoying if they shut down the system in the middle of an RPM Install. (one dual boots to Windows so he's more likely to reboot, the other runs solely on Linux he really only powers off if he's heading out. I think I've installed one or two kernel upgrades in the last year (which require reboots to enable), but since my roommates reboot so often, I can just wait for their next reboot.
There's also much less need to do testing with Linux patches... You generally know EXACTLY what subsystems are being affected by a patch, so if it's not a critical component, you can often install blindly. Even if it is a critical component, the patches are often well defined and if you have any questions you can read the source code.
The problems with Windows is that it's the large-scale version of spaghetti code. The relationship between various pieces are ill-defined and numerous. Patches spider into various areas and it seems like nobody (even at Microsoft) knows precisely what a patch fixes (or what it breaks).
This doesn't just apply to desktops. I'm in the middle of putting together scripts to enable controlled push of patches to a large number of varied servers. In truth, the hardes part is going to be figuring out which patches go to which boxes -- not figuring out if the patch is going to break things.
Yep. I'm spoiled. Linux makes life both easy and cheap.
Let Your Customer Decide (Score:3, Insightful)
So, we changed the contract. Unscheduled downtime projected to last more than 30 minutes required getting permission from several designated management types. Any one of those managers could postpone the maintenance.
This worked because the support contractor always made sure that those designated managers understood the implications of delaying the maintenance.
Real examples of why its sometimes good to wait (Score:3, Interesting)
Twice a month, we release patches which fix any number of bugs we may have found since the original release of the software. About 1/3 of the patches we release introduce NEW bugs that weren't there before the patch! These new bugs can easily and often cripple important parts of the software.
I knew a 4 month stretch where this happened on every release for those 4 months, 8 patches in a row!
Most of our customers update every few months, and they keep an eye on our website, and the public customer email lists constantly throw out emails which the bleeding edge leaders complain of problems introduced on new builds (which they have every right to complain about).
Now I can't speak for any other company, including Microsoft, but sometimes upgrading right away when you aren't really currently experiencing an active problem is worse than not upgrading at all.
What is this rebooting of which you speak? (Score:3, Funny)
Hm, rebooting. Rebooting. Oh yeah, I remember now. I had to do that to my GNU/Linux system once when I upgraded my motherboard.
1 day (Score:3, Funny)
Patches? (Score:3, Funny)
Just run a VAX/VMS system as your firewall... it's so old and obscure that no hacker will have any hope of remembering how to hack it.
Re:Throw caution to the wind (Score:2, Informative)
Re:So? (Score:2)
I too am a consultant. I do *not* get the right to tell them, "You are going to do this. You are going to do it now. And you are going to pay me for it." It's their money, it's their company. My job is to present them with options (and one option is *always* to do nothing), and make them aware of the consequences.
I make that point as strongly as I can, but in the end, I do not have the right to spend their money for them.
As much as I'd like to, when the client who *just* had