Why Is Less Than 99.9% Uptime Acceptable?

Ian Lamont writes "Telcos, ISPs, mobile phone companies and other communication service providers are known for their complex pricing plans and creative attempts to give less for more. But Larry Borsato asks why we as customers are willing to put up with anything less than 99.999% uptime? That's the gold standard, and one that we are used to thanks to regulated telephone service. When it comes to mobile phone service, cable TV, Internet access, service interruptions are the norm — and everyone seems willing to grin and bear it: 'We're so used cable and satellite television reception problems that we don't even notice them anymore. We know that many of our emails never reach their destination. Mobile phone companies compare who has the fewest dropped calls (after decades of mobile phones, why do we even still have dropped calls?) And the ubiquitous BlackBerry, which is a mission-critical device for millions, has experienced mass outages several times this month. All of these services are unregulated, which means there are no demands on reliability, other than what the marketplace demands.' So here's the question for you: Why does the marketplace demand so little when it comes to these services?"

because they've been conditioned

Why does the marketplace demand so little when it comes to these services?

The marketplace has been duped into believing that this is the best technology can provide. People don't have time to know, understand, or research history and find that technology really can be reliable.

I'll get modded troll, but I lay much of this at Microsoft's feet. I laughed them off when I first heard of them and their goal of taking over the industry. After all, I'd been working on systems that ran 24x7 with five-9 reliability for years, and DOS/Windows couldn't touch that.

One time I had an opportunity to visit Microsoft and have lunch with a friend there. I figured while there I'd take the opportunity. I asked them in hushed tones, "Just how do you configure Windows so that you don't have to reboot it all of the time?" They looked at me like I was crazy.

Technology can provide reliability. The general public is no longer even aware that it's possible.

Re:because they've been conditioned

The reasons why Microsoft were so successful (in a business sense) are manifold, but one is not that their products were great, but that they were good enough. They accurately measured what people would put up with at different price points, and serviced the market accordingly. I think ISPs, telcos, etc have done likewise.

At what price?

"The marketplace has been duped into believing that this is the best technology can provide. People don't have time to know, understand, or research history and find that technology really can be reliable."

No. They believe it is the best the technology can provide at a given price. Why do people "put up" with cars that only give them X amount of protection in a car crash even though there is technology out there that would make them safer? Because they aren't willing to pay the marginal cost for the extra protection. Arguing about what is possible with technology is pointless. What matters is what a piece of technology can do at a given price.

Everything is a trade-off. The sooner Slashdot learns this the less we will have these stupid "Why don't consumers use the latest, greatest, most expensive technology? We need to force them somehow!" articles.

Re:because they've been conditioned

I think the term you're looking for is "managing expectations." Here's a little article about it from the IT side. [ittoolbox.com] It's something that Microsoft and teleco's have become so good at. If you keep expectations low and give them a little better, they'll be more than happy. If you give the same, but you promised the world, you get a bunch of unsatisfied customers.

Re:because they've been conditioned

One time I had an opportunity to visit Microsoft and have lunch with a friend there. I figured while there I'd take the opportunity. I asked them in hushed tones, "Just how do you configure Windows so that you don't have to reboot it all of the time?" They looked at me like I was crazy.

In a certain sense.. you were crazy, at least at Microsoft.

The origins of an OS really show through a lot of the time. Windows started out as a single user OS, so rebooting was OK because the only person you messed up was the guy sitting in front of the screen. It eventually evolved into a multi-user OS, but the "just reboot!" mentality persists to this day.

Linux/Unix on the other hand started out life as a multi-user OS. Rebooting was a big no-no, because you'd affect countless people logged in, and you'd get yelled at for ruining someones work.

It's funny the attitude that comes from the users of each OS. Windows administrators categorically will try rebooting the damn thing first to fix any problem (and it usually works). Linux administrators will only try this as a last resort (and it almost never works).

Anyway, at Microsoft the idea that you can somehow tweak windows just right so rebooting isn't necessary is crazy. They designed the damn thing so "just reboot!" will fix any problem. This of course is an unacceptable solution to a lot of people out their, but for a lot of people it's obviously reality.

Re:because they've been conditioned

Conditioning certainly has to be a big part of it. People put up with crappy wireless phone service because that they don't remember (or are too young to know) what an old-fashioned fully-wired telephone conversation sounds like. After a couple decades of cordless and wireless phones, the level of service has gone from "you can hear a pin drop" to "can you hear me now?"

No way...

This has everything to do with cost and nothing to do with Microsoft. Consider VoIP... people are deliberately choosing telephony services that are less reliable and lower quality than POTS, because VoIP is cheaper. If you want 99.999% uptime, that's fine -- but you're going to pay for it. High availability services require better equipment, redundant equipment that doesn't come cheap and more, higher quality staff to operate it. So it costs more.

I've been in the technology services business for a long time, and with few exceptions, 80%+ customers want their services are delivered as cheaply as possible. Most hospital systems don't even have a 99.999% availability requirement. The 20% the want varying levels of higher than normal availability usually have a government regulation, SLA or other mandate requiring that they do so.

Reality Check

In fact, though, I can tell you that in the pre-Windows days, electricity had outages, television had outages, telephone service had outages, gas service had outages...

I was born in 1964. I have no recollection of POTS telephone service ever being unavailable.

Electricity was expected to drop out a few times every summer, and until someone figures out how to tell lightning where to go, I expect it will continue to happen. In my part of Canada, however, power is continuously available from October to April no matter what. Even if you don't pay your bill. The only winter power outage of note I can think of offhand was the great Ice Storm of 1998 [wikipedia.org], one of the most spectacular cases of force majeure I've witnessed in my life.

In my part of the world, at least, power and telephone were life-and-death services and legislation mandated their reliability.

Re:because they've been conditioned

The server is up 99.99% of the time. The server's T1 is up 99.99% of the time. T1's ISP is up 99.99% of the time. The backbone provider is up 99.99% of the time. The cellular ISP is up 99.99% of the time. The cell-to-tower linkage is up 99.99% of the time...

Eventually, with all of these little points of failure, you're going to get a good sized chunk of fail. Add in things like the inherent instability of wireless technologies and our nation-wide problem with an aging electrical infrastructure, and you have the sorts of occasionally mildly-inconviencing issues that you see today.

Right now it seems like the things users want to optimize most for are A: speed and B: cost. One day every other month where our home internet is down doesn't seem like the end of the world, especially with the cost of the alternative.

The cost

Probably because of the cost. I do network design for a fairly large telco, and let me tell you the cost goes up exponentially with the number of "9"s that the business asks for.

Re:The cost

Exactly what I was thinking. I work for a CLEC, and I have a rough idea how much things cost -- compare what a Lucent 5E costs with what a top of the line Cisco router costs, and you have the answer why voice service achieves five-nines while data service typically does not.

Bingo

It's all about cost vs. the cost of downtime. You'll find in business lines such as the financial sector, customers are willing to pay for extremely high availability because time is indeed money. Business lines that have lower costs for downtime have to weigh availability vs. ROI.

Re:The cost

Also, you need to bear in mind that POTS is incredibly simple technology compared to Internet/Cellular/Data services. I haven't had cable TV since the early '90s, but I don't ever remember it going out, either- that was long before we had digital cable/cable Internet in my market area. POTS never goes down because the equipment is extremely robust, even (especially) the older stuff. My local telco could continue to provide POTS for more than 4 days during power outages simply because of lower power requirements (after 4 days, they had to fire up their generators, and started dropping remote COs due to extreme cold).

We always want to compare service levels for newer tech with POTS and complain when they don't approach the same levels, but I'd expect that if we were to be still using the same equipment for ISP/Cellular service in a hundred years, it would be as stable and robust as the current (ok, previous generation) iteration of POTS. Problem is, we are constantly demanding better, faster, and cheaper: this has to be traded off for reliability, and for the most part people are happy with that tradeoff. Just like we're happy to buy crappy consumer goods from China at Wal*Mart because they're cheaper than domestic products. /rant

Here's an easy one.

Quoting the summary:

... after decades of mobile phones, why do we even still have dropped calls?

It's a little thing called physics. When you're traveling while using your phone, you may transit into dead zones [wikipedia.org]. We could solve this by cutting down all the trees and flattening the landscape, but that might make some people angry...

Re:Here's an easy one.

As a guy who does communications in the U.S. Navy, I can attest to this. If the United States military can't guarantee 99.999% uptime on communications in all conditions, what makes anyone think it's possible in the private sector?

Low price or high-quality?

You can have one or the other.

We're not talking about software, we're talking about hardware and man-hours. Those will never be free.

Really so common?

Are these kind of outages really so common? Mobiles phones I absolutely agree with. ON the other hand, I literally cannot remember the last time I lost cable or my internet. I've literally lost power more frequently than either of them (maybe 4 times in the past year) and lost water once. Emails not making it to their destination--again, does this really happen? In the decade plus I've been using internet email, I can't off the top of my head ever think of any "lost" email unless it was sent to a wrong address or something.

because 'misson critical' is a myth

Because 90% of stuff labeled 'mission critical' actually isn't. Think about it - for most of us, being able to receive or send cellphone calls or emails at any time seems super important, but the number of hours in any given month where it really *was* super important (the grant application was due in two hours; your mother was sick; your partner was about to go into labor; whatever) is generally pretty low - our real tolerance for occasional downtime is therefore quite high.

You don't have to take it anymore

When Comtrash Internet dropped my speed from 6 Mbps to 1 Mbps but kept the rate at 6 times DSL, I dropped Comtrash and went with the 1.5 Mbps DSL from my local telco. I got 50% more than Comtrash was delivering at 1/6th the cost. No problem.

When Microsoft decided that I didn't own the rights to my own media and stopped me from being able to copy my own DVDs, I decided to drop them for my media development system and I switched to Linux and Apple. Microsoft doesn't want my business so I went with the people who do. No problem.

When my Long Distance company decided to charge over $1.00 per minute for International calls, I switched to AT&T and their 17 cents a minute program. No problem.

When Frigidaire washers charged extra for the warm water cycle but only give you 5 seconds of hot water and thus, never any, it was no problem to return the unit and buy a different brand. Sure, the salesman wasn't happy but, that is now his problem and not mine. I bought a different brand that did give me what they advertised and promised. No problem.

The list is endless and across all businesses and domains.

The point being is that there are alternatives but, many (or most) people are either too lazy to do anything about it or, like this article, they are too apathetic to do anything about it.

The choice is up to the consumer and, if the consumer would take action, the industry would have to adapt because the market demands it. So far, the market is willing to accept this and thus, the industry sees no reason to change. The less the consumer will accept for their dollar the less they will receive. That, is the problem.

O RLY?

When it comes to mobile phone service, cable TV, Internet access, service interruptions are the norm -- and everyone seems willing to grin and bear it: 'We're so used cable and satellite television reception problems that we don't even notice them anymore.

And television is mission critical? Besides, I bet most people don't experience significant cable TV interruptions. Satellite depends on the strength of the signal. Tap into Arecibo and you'll likely get 100% reception.

We know that many of our emails never reach their destination.

[citation needed] I call bullshit on that one.

Mobile phone companies compare who has the fewest dropped calls (after decades of mobile phones, why do we even still have dropped calls?)

Because it's a benefit to have a phone that doesn't draw so much power that your brain heats up just from using the device. Also, dropping a call indicates that you're in an area where there's no cell towers or because you've hopped from one tower to the next and the next tower has its connections maxed out.

And the ubiquitous BlackBerry, which is a mission-critical device for millions, has experienced mass outages several times this month.

Blackberry is not a mission critical service. The people who use it as such are naive. If there truly is a market for five nines uptime for Blackberry, RIM would develop such a service and charge an order of magnitude more for it.

All of these services are unregulated, which means there are no demands on reliability, other than what the marketplace demands.' So here's the question for you: Why does the marketplace demand so little when it comes to these services?

Because ultimately it's really not a big deal. So your satellite TV goes down for a bit... get a life. You drop a cell phone call... redial. Your Blackberry isn't receiving emails... get a life.
 

WTF would I do with 99.999% uptime?

My electricity isn't 99.999% uptime (that's 30 seconds in a year) which would require me to get an UPS
My consumer grade equipment isn't 99.999% uptime (with luck, maybe I guess but there's no ECC, redundant power etc).
My software isn't 99.999% uptime (ok, so the kernel is stable. When X crashes, so does everything of importance on a desktop)
If there's something urgent, you CALL me anyway.

I'd rather take a line with 99.5% uptime (that's two days without internet per year) that's 10x faster and costs 10x less. Which doesn't include that I have Internet at work, or via my cellphone, or via a webcafe or any number of other easily available sources. The only real killer I can think of is if you only telecommute and can't go to work, but even then I figure the nearest Starbucks will let you occupy a corner with some purchases.

Re:Costs increase geometrically

Because every nine will cause a geometric increase in costs.

This

Uptime (%) Downtime 90% 876 hours (36.5 days)
95% 438 hours (18.25 days)
99% 87.6 hours (3.65 days)
99.9% 8.76 hours
99.99% 52.56 minutes
99.999% 5.256 minutes
99.9999% 31.536 seconds

I work for a software shop where we can do high availability, but more often than not, folks chose to lower the uptime expectation rather than pony up for the stupid money it takes to have the hardware / software / infrastructure to get there. Most companies know the customer will not pay the extra cash for the uptime, thus... you get what you pay for.

Re:Costs increase geometrically

My concept of 5 9s is much easier: 9.9999%. Or for Vista servers, .99999%.

Not So Simple

To put it simply, it's the money stupid. It requires a lot more equipment and manpower to offer a high availability service. This extra cost results in higher prices. It can cost 1000% more a month for less than 1% more reliability. Think of a $400 a month T1 with a SLA versus a $40/month cable line. Being sheep has nothing to do with it.

Re:The way it has always been

While you deserve the mod points, it should also be noted that consumer expectation is strangled into submission within 20 minutes on the first support call they make to ask about better service quality.I know a guy who is locally famous because he will spend 4,5,6 or more hours on the phone with customer service, supervisors, managers and anyone on the board of directors that he can find a phone number for. What is he fighting for? discounted service or reparations for lost service(s). That's right, it takes hours on the phone to get one of those companies to either own up to, and pay for losses accrued by their customers through loss of service.

In truth, most consumers won't complain when they should, so there is no marketplace pressure on those businesses to aim for five nines uptime.