History of Software Patches?

NinaBeth asks: "I'm interested in the history behind software patches for an academic paper I'm writing. In particularly, I'm wondering what motivates shrink-wrap software companies to release patches? Why send out 'broken software'? Is it purely financial? Has anyone done a cost-savings analysis of QC prior to release versus user-reported problems? Any stats on the average number of patches an application will require? Is any one particular company more patch-happy than others? I don't need much, just a reference or two would be helpful. Thanks for any suggestions!"

Why?

[hogsback]

Why send out 'broken software'?

Usually because you don't know it is broken at the time. The real world is a much harsher test environment than internal or even beta testing.

Re:Why?

[larien]

The real world is a much harsher test environment than internal or even beta testing.

As per the The Ten Commandments for C Programmers [lysator.liu.se] says:

...for surely where thou typest "foo" someone someday shall type "supercalifragilisticexpialidocious"

It's amazing how silly/malicious some (l)users can be.

There's also the maxim that companies keep writing more foolproof software, but the world keeps building better fools.

Re:Why?

[dreamquick]

Simple.

Let's look at the computer games industry, they who made releasing patches a religion, plus their model closely follows the more generic model of patches in that they will get issued when it either makes the manufacturer money or when the product is getting bad press because of defect X.

Scenario #1:

Project XYZ is months beyond the deadline, it would cost too much to scrap the project, so slap on some "eye candy", tidy up the rougher edges and we could sucker people into buying it on brand-recognition-value alone, not to mention the hype we generated over the last three years while it was under development.

Once its out it generates much needed positive revenue (as opposed to negative revenue) and we can patch all the left over bugs out of it, or add the extra features which were stripped to get it ready.

Scenario #2

Despite your best efforts your game becomes an overnight success, but the network portion of the game was tested at a much lower level of stress.

By releasing a patch you ensure that more people buy the game and play it online there-by increasing your revenue flow.

Market exposure worth more than bug fixes?

[marcsherman]

One theory is that early market exposure is worth more for a product than fixing the bugs with a patch will cost. Every day a product stays out of the market, it's losing 100% of new sales on that day to the competition, and down the road when it's time to upgrade, people are much more likely to stick with a product they already know than to switch to another brand.

Just part of software.

[Xenopax]

To answer your question on if patches are finacially motivated: yes and no. Yes as in you have to release a product eventually, but no because you will never write a perfect software program the first time around. There are so many things that can effect your software from hardware, drivers, other software, etc. You can not possibly test every single configuration out there, so at some point you have to rely on feedback from your customer to find problem areas.

That being said, there are times when products are rushed out the door before they are ready. I had experience with that at my old company, and the reasoning was purely finacial. I really can't fault them for releasing early though since we had to play a delicate balance of releasing something good and going under, and eventually had to split the middle.

patch early and often

[fonebone]

my operating system [kernel.org]'s been releasing patches about once a month since i started using it, and it doesn't look like they're about to stop!

Funcom vs. MS

[JMZero]

The wording of your question makes it very easy to think of you as a troll, though you're probably not.

A good example of a company rushing software is Funcom's Anarchy Online. It was released in spite of huge, unresolved problems with the beta tests. However, they felt they had to go to market ahead of competitors.

Other products are simply doomed by diminishing returns. An alpha of a new MS OS might have tens of thousands of bugs. Internal testing will get a lot at first, but the bug finding rate will wind down quickly as time goes on. After a certain point, it's more economical to let the public find the bugs than it is to hire more testers.

There will always be some equilibrium here, though I'm guessing MS will likely try to do more testing in future releases - as the talk of insecurity is actually starting to get through to the general public.

Software Patches

[Slipped_Disk]

Well I can't say it authoritatively, but patches have probably been out since the very beginnings of programming - at the very least since the beginning of UNIX.
When you think about it, all the Unices, commercial or free, have always released patches. In Linux and the open-source BSDs these are released as source code (diffs or checked out of CVS), in the commercial world binary patches that either replace or edit a portion of the files on your computer are often released to address security or functionality problems. I honestly can't think of any other major piece of software (OS, app suite, windowing system) that hasn't released at least one "apply this today or your computer may explode" patch.

As for the "why" of releasing broken software, I can personally attest to the fact that most companies probably DON'T know about the problems until they come up in the real world. When my company tests software we try to think up extreme or improbable cases along with the mundane, but invariably we miss something.
IMHO the releasing of buggy software isn't necessarially bad - but on the other hand if you KNOW a bug exists and it is fesable to resolve before a release, that should be the prefered solution (as opposed to a patch later).

As for the average number of patches a piece of code requires, the codebase for the larges application I am currently working on has had well over 100 internal patches (things which didn't cause functionality problems but still should have been fixed). These fixes were sent to customers in 10 seperate external patches (a patch that increments the z in x.y.z versioning) which also fixed functionality problems that we discovered in additional testing or that the users reported.

For larger-scale examples, check out the lists of patches for solaris (www.sun.com) and MS Windows (windowsupdate.microsoft.com - assuming it's back up)

Hope this helped a little.

Re:Software Patches

[SuiteSisterMary]

Sun is pretty cool because they patch their patches; you'll notice that every patch has a revision number. When I was doing tech support a few years back, we had a few Solaris customers call in. They were getting segfaults on execution of our product. Nobody else. Just three or four people. We checked hardware, we checked the patches we required, we checked other running software, we checked everything. Until, one day, while I'm staring at the Sun webpage for one of the patches our product required for Solaris, one of the customers, with whom I'm reviewing everything we've done up till now, says something like 'And the patch 238509 rev 5 was installed...' and the page I'm staring at says '238509 rev 6' with a release date of a week beforehand. Grabbed an archival copy of rev 5, and sure enough, it was available for something like a week, and caused the segfault.

Re:Software Patches

[cymen]

No offense but why is this cool? If Sun would just bump the patch version your whole experience with versions of versions could would probably have been avoided.

Re:Software Patches

[briany]

This is cool because you don't have to remember that patches 294845 and 392843 and 492835 and 420293 are kernel patches. You just remember that patch 452593 is the kernel patch for Foo version 2. Incremental patch revisions provide better fixes, or similar fixes that show up elsewhere.
Makes a hell of a lot of sense once you get used to it.

Re:Software Patches

[cymen]

Yeah, I can see that. I wonder if the versioning system Microsoft uses is bad. They tend to seperate out different increments/things with a "." so you'll have 444.232.131.23, etc... It too is kind of hard to read at first but personally I'd prefer it over something like 444 rev 23 (444.23 is more readable to me).

Re:Software Patches

[briany]

The numbers are 12345-1 for rev 1, 12345-2 for rev 2, etc, etc. The dash helps keep the confusion down; periods sometimes get lost in frantic searches for the right patch.

Re:Software Patches

[cymen]

OIC. From your original post I came to the conclustion that the reason the revision was overlooked was that it was seperated from the version number. A dash makes sense.

Er... Now we rewind two iterations and continue on, hehe...

MS-DOS

[Suicyco]

I dont think MS-DOS ever had any patches, just new versions. (not a troll, just an observation..)

Re:MS-DOS

[zenyu]

It had plenty of bugs though. I don't think we really used in programs except for disk access, and often that was done through BIOS calls or just direct access to hardware. Of course, that meant an app might only work with IDE disks, two brands of mice and three video card pseudo-standards. And the apps did have patches, though I think not having the internet to distribute them was a major factor. Plus, security patches aren't as important when you need physical access to the machine to do any harm.

Remember you took out the DOS floppy when you loaded the app, there is a limit to the number of bugs in 8k of code ;)

all you really need to know about patches...

[msouth]

...is that sooner or later we'll see a patent on it. "Method for releasing small incremental change file to propagate fixes in large distributions." or something like that.

:)

Re:all you really need to know about patches...

[Tower]

Actually, Symantec has a similar patent for their virus/scanning engine updates... who is to say what "large" means... 640k was "large" once.

Re:all you really need to know about patches...

[Servo5678]

Be quiet! We don't want to give anybody any ideas...

They Write The Right Stuff

[yancey]

You should refer to the article http://www.fastcompany.com/online/06/writestuff.ht ml [fastcompany.com] entitled "They Write The Right Stuff" for some information about the developers who write the software for the Space Shuttle. It partially addresses your question.

"Most people choose to spend their money at the wrong end of the process," says Munson. "In the modern software environment, 80% of the cost of the software is spent after the software is written the first time -- they don't get it right the first time, so they spend time flogging it. In shuttle, they do it right the first time. And they don't change the software without changing the blueprint. That's why their software is so perfect."

Re:They Write The Right Stuff

[gorillasoft]

From the article: ... the groups $35 million per year budget is a trivial slice of the NASA pie, but on a dollars-per-line basis, it makes the group among the nation's most expensive software organizations.

I don't think following their method is a viable commercial model. You would virtually always be beaten to the market, and you would have such large up-front costs that you may not be able to survive until the software is released and selling well. NASA has the benefits of massive funding and defined problems that they can wait to solve correctly rather than worrying about selling a product just to survive. Oh, and if they don't do it right, somebody may be killed. That's a large incentive to spend the time and dollars necessary for nearly flawless software, but it doesn't translate to most corporations' markets - thus the release of buggy software that requires later patching.

Deadlines, quotas, metrics, and some history

[imrdkl]

First, if you are really interested in the history of patching software, then check out the patch(1) program for applying source level patches. This was written originally by Larry Wall, also the author of perl, and a well-known cofounder of the internet (so to speak).

If you dropped Larry a line, I'd bet he'd be willing give you some perspective, at least from the Free Software point of view. He used to release patches directly on USENET, now that was a great way to keep customers happy...

The rest of your question can probably be answered by the first three words in the subject of my comment. If it builds and runs on the announced shipping date, and there are no (or very few) show-stopper bugs, then most companies will cut CDROMS and ship.

Why we patch

[cassidyc]

I work for a software firm that produces safety critical distributed systems. We have close relationships/support with our customers, and will, if the customer finds a safety critical issue, provide a patch.
Non critical issues, however, get fixed in subsequent upgrades/releases, as a rule, we tend not to patch these.

We don't release "broken" software, but real life usage finds thing that we just cannot test for.

The "Capability Maturity Model" (CMM)

[Ocelot Wreak]

One direction that has picked up a lot of credibility and support from large software developers (for big .gov and .mil projects) is the Capability Maturity Model (CMM). It has been applied to software developement for applications, computer security engineering, etc. CMM allows the software development process to be more closely managed, observed, measured, and fixed _before_ it breaks, or where errors can cause big problems in critical systems. Several good books have been writen on it, and a lot is available on the web. See:
www2.umassd.edu/SWPI/processframework/cmm/cmm.ht ml for some general background on the process. From their web site:

SEI Capability Maturity Model

The CMM describes the principles and practices underlying software process maturity. It is intended to help software organizations improve the maturity of their software processes in terms of an evolutionary path from ad hoc, chaotic processes to mature, disciplined software processes. The focus is on identifying key process areas and the exemplary practices that may comprise a disciplined software process. The maturity framework provided by CMM establishes a context in which:

Practices can be repeated, if you don't repeat an activity there is no reason to improve it. There are policies, procedures, and practices that commit the organization to implementing and performing consistently.
Best practices can be rapidly transferred across groups. Practices are defined sufficiently to allow for transfer across project boundaries, thus providing some standardization for the organization.
Variations in performing best practices are reduced. Quantitative objectives are established for tasks; and measures are established, taken, and maintained to form a base-line from which an assessment is possible.
Practices are continuously improved to enhance capability (optimizing).

Structure of CMM
Maturity Levels
A layered framework providing a progression to the discipline needed to engage in continuous improvement (It is important to state here that an organization develops the ability to assess the impact of a new practice, technology, or tool on their activity. Hence it is not a matter of adopting these, rather it is a matter of determining how innovative efforts influence existing practices. This really empowers projects, teams, and organizations by giving them the foundation to support reasoned choice.)

Key Process Areas
Key process area (KPA) identifies a cluster of related activities that, when performed collectively, achieve a set of goals considered important.

Goals
The goals of a key process area summarize the states that must exist for that key process area to have been implemented in an effective and lasting way. The extent to which the goals have been accomplished is an indicator of how much capability the organization has established at that maturity level. The goals signify the scope, boundaries, and intent of each key process area.

Common Features
Common features include practices that implement and institutionalize a key process area. These five types of common features include: Commitment to Perform, Ability to Perform, Activities Performed, Measurement and Analysis, and Verifying Implementation.

Key Practices
The key practices describe the elements of infrastructure and practice that contribute most effectively to the implementation and institutionalization of the key process areas.
Good luck!
-wjc.

Dear Slashdot!!

[Wakko Warner]

I have a paper to write, and I'm too damned lazy to research it. Can you guys do it for me? Please be as specific as possible, and try to include full references if you can.

Thanks for all the help!!

- A.P.

Re:Dear Slashdot!!

[Wakko Warner]

Actually, my normal argument is that the person should've used a search engine [google.com]. In this Internet era, I am continually amazed at the complete lack of experience in this area, even on slashdot.

- A.P.

Re:Dear Slashdot!!

[Fiznarp]

Oh heck, with all the self-appointed geniuses here, why waste your time clicking links from a search engine?

I do all my homework on slashdot... well, except for sex ed. I do that on Everything 2.

:)

Fiz

You should probably be taking a business class...

[xarc]

Not that I am a business major, but patch levels make sense to me.

Time is definite issue. First of all, you want to get your product out the door.

Second, you want development to be as fast out of development and as feature rich as possible. This doesn't always allow for perfect code.

Third, you don't want your team to burn out.

Therefore, most (smart) software companies will release patches to their software. Also, releasing patches gives the consumer the impression that you are actually maintaining your code (whether you are or not).

Think about trying to write Windows XP or KDE or GNOME from the ground up-- in one release. Not going to happen, unless you have a lot of dedication and a lot of time. And by the time you finish, it might well be outdated, or even unliked.

On that note, an additional benefit of releasing software in patch levels (or SPs), is you get a large showing of customer feedback. If there are major bugs, they will be found. If your software is "good enough", you might be willing to distribute it. I suppose that's what makes Dilbert's "It compiles! Ship it!" so funny.

Granted, this is all opinion.

Here's a few.

[ReluctantBadger]

"Is any one particular company more patch-happy than others?" In my professional experience, I have found that two names stand out as the "patch kings". Not to Microsoft-bash, but there is a good deal of patches released for nearly all of their software and components. You may want to try looking at recent patch histories at Microsoft for IIS, and critical updates for Windows 98 and Internet Explorer. I spend maybe 2 days a month doing a full probe and catalog of the knowledge bases and product areas looking for updates, especially for Outlook. My other favourite is HP. Out OfficeJet T45 needs new driver pack downloads with every release of Service Packs for Windows 2000. The original Win2k T45 drivers worked OK on Vanilla and SP1 systems, but when you upgrade to SP2 - BANG! 100% CPU usage. This is just one example of many. Their VL400 series and Brios have had many, many revisions and updates. One of the really annoying things with HP Brios is the vast difference in the product model numbers and video/chipset drivers. Most irritating, especially when I want to do a GHOST rollout. Yes, this is hardware, but DAMN - it's annoying. No doubt here come the "Then why are you using MS?" and "why not standardise your user desktops for GHOST rollout" replies. Fact is, in the real world of support, it isn't always that simple. I am trying to get everybody on the same VL400 workstation, but you gotta justfy the cost savings against IT support time!

Patch Crazy or Crazy Like a Fox

[thumbtack]

I've had experience in purchasing software that had to be "patched" as soon as it was installed. I'm not talking about software that's been setting on a shelf in a retail outlet for 6 months but bought straight from the manufacturer, and delivered next day. Upon installation, I had to go to the website to check for updates and patches, which there always were. Of course, only registered users could get them, so it seems to take on a whole new light. Rather than patches were they actually verifing the registration info and that I wasn't one of a million that were usng the same serial number?

This has happened with different manufacturers on several occaisions, so it wasn't an isolated incident.

Behind it all...

[vukv]

Basically, your question is why cant software be developed to perfection?

Basic reason is that we always keep adding new features to it. It is impossible to stay on one set of features through product life. You simply have to add more and more and more... and you will never be able to test it properly. Not to perfection.

However, bigger issue lately is that companies are short on cash so they have to release software that is not ready to be released - software that has not been properly tested. Example I... World War II Online game - absolutly bad piece of software, kept crashing, etc... after several patches, its one of the best multiplayer games around... Example II. Mac OS X - first release did not have full support for all Mac hardware you could purchase (notably cdrw/dvd recorders)... they simly said we need to get it out and add this later

Most software that gets released actually deserves early beta status, which is why you dont see companies upgrading until at least 2 patches are released.

And even if you freeze your feature set, it cant last long... soon enough competition will have you scrambling to add more features and you wont have a year to properly test it... simply, I doubt there was ever a piece of software that was perfect and I doubt it will ever be... I for one cant wait for next patch for my router, OS, game or my favorite software


:P

Software Reliability Engineering

[arb]

You might want to check out some books on software reliability engineering which would help answer your questions. One in particular I have read is simply called "Software Reliability Engineering" by John Musa.