Grading Telco & ISPs During the Blackout of 2003?

alt_cognito asks: "Our company runs natural gas generators here in Novi MI and when the power went out we didn't miss a beat. Nine hours later, our telco blinked and our T1 service went down despite the lines being run to different locations and ISPs (UUNet, LDMI). Service did not return until power had returned to the upstream offices. I was under the impression that these locations would be run by similar power generation. How did your telco/ISP perform?"

Re:K5

[Creepy Crawler]

Oh yeah. I've always had minimum pingtime of about 30 seconds to render a damned page. And I'm on DSL.

Seems scoop isnt all what they thought. Guess they shold have chosen slashcode.

Dial Up and DSL

[J2000_ca]

I had both dial up and dsl through the whole thing.

Re:Dial Up and DSL

[mini me]

As did I. I know because I was running my computer off of a generator and it was working just fine.

Escape from New York

[AtariAmarok]

In flyover country, the only online problem I had was one banking site that had a stuck screen. Everything else worked fine, if not better (since some sites were much less overloaded with Northeasterners).

If the problem originated in Cleveland, I still think the problem had to do with Lewis and Oswald.

Re:Escape from New York

[sharkey]

If the problem originated in Cleveland, I still think the problem had to do with Lewis and Oswald.

Hmmm, bet they let some of the test subjects out.

Same situation

[Captain Rotundo]

Our generator worked fine (we were powerless till about 5am friday) and a dedicated line to Motient stayed up the entire time. But our UUNET T1 was down until about 6pm friday. and the entire time it was impossible to even get a Worldcom person on the phone to find out if they even had an idea as to when they would get thier act together. so basically I give Motient an A+ (well they were outside the blacked out area but I still got updated on thier network status inside the area) and I give Worldcom a big fat F.

Re:Same situation

[photon317]

I give WorldCom a big fat F too, right up their asses, they suck.

Re:Same situation

[amackinlay]

Strange, UUnet/WorldCom/MCI were 100% with no lose or degredation for us. Though I'm talking data center colocation.

Re:Same situation

[photon317]

Back when I used to work at an ISP (like back in 1994), I was a big MCI fan because they were way more reliable than Sprintlink. My bad feelings about them mostly stem from them laying me and my whole department off (I had been there in a development-related position for 4.5 years) when they had some financial troubles due to mismanagement. Makes you think when they whack 100 or so of their brightest employees without blinking to save some cash that their executives blew.

Just fine

[linuxwrangler]

My ISP handled the outage just fine. Here in California we don't have to worry about blackou&*#%@

carrier lost

Re:Just fine

[Fareq]

Indeed, here in California, we have reliable power in limited quantities at high prices.

We (or at least I) pay > $0.13/kWh
If the state uses too much power we get rolling blackouts.

But, actual power failures are quite rare (and brief) at least in my experience.

Anyway, back to the topic at hand:
my ISP (Adelphia) was up the entire time. The internet in general seemed somewhat flaky, with lots of random things running very slowly or not at all.

Of course, I would have been shocked if a California provide

Sympatico fine

[mnmn]

Here in Canada, Bell Sympatico was up from when I lost power to when I got it back, I would just assume they were up all along. All ISPs I believe have failsafe setups that were never tested and maintained and that showed very well during the blackout.

At my company the IBM eSeries servers were backed by a smart UPS that showed 17 hours remaining, 15 seconds before shutting down everything in cold blood. It was all scandisks booting back up there on Windows 2000 machines.

I maintain some small servers with no UPS in a few locations, and while one Solaris server crapped out, you had to manually do the fsck thing, all the FreeBSD servers were back up as the power came without a hitch. I have to learn to setup Solaris on sparc so it fsck itself without asking for an input.

Re:Sympatico fine

[Anonymous Coward]

Yeah well, here out east in the Maritimes ALIANT, who make themselves out to be a big maritime company, had their mail server offline for nearly an entire day because... it's in toronto. There's an "oops".

"What's that you say, you claim to be a maritime company supporting maritime jobs but your stuff is all in another province?"

Verizon is vucked...

[Stardate]

...and without a contract, I doubt the workers will be working very hard to fix the damage caused. I have multiple T-1 lines down all over Manhattan, most on UUNet/WorldCom. The word is Verizon has a couple CO's that are still out, and I hope they know what they're doin....

What does that have to do with a blackout?

[sharkey]

Nine hours later, our telco blinked

Since your telco is SBC, that should be a daily occurance.

Re:What does that have to do with a blackout?

[funaho]

SBC is absolutely horrible. Since they bought Ameritech the service in the Detroit are has gone through the floor. I have two T1s in the city, and both have a 75% chance of going down for a day when we get a good hard rain. In fact one of them was down literally every week for six weeks straight, until they finally messed around with the aerial cable a bit.

I've had more than one repair guy tell me that when SBC came in they changed the rules that govern when they replace bad cabling. I think all 9 planets

Bet your local telco is the problem.

[FreeLinux]

It's great that you have two different providers. Eliminating the single point of failure is important. But, most people miss the semi-hidden single point of failure, the local telco. The problem is that your two ISPs likely don't own the copper that those T-1s run on. That copper is owned and operated by your local telco. Your ISP just contracts with the telco to provide you their service over the local telcos loop.

It is likely that you will never find out exeactly what happened but, from what you describe it sounds like; the lights went out, the local Central Office(CO) where the local loop for your T-1s went onto UPS backup or generator and after a few hours the UPS or the generator ran out of juice. Once the CO ran out of juice your T-1s went dead. So, you lost connection with the ISPs. More than likely the ISPs themselves never blinked.

The only way to avoid this problem it to use two different local loop providers which is usually going to be hard to find unless you are in a large metropolitan area. The other thing to do is get the local loop lines from different COs which will be like pulling teeth from your local telco.

Planning and preparedness, unfortunately, does not guarantee against failure.

Re:Bet your local telco is the problem.

[Myself]

I'd bet otherwise. The local telco's CO has 18+ hours of battery and a week of diesel/kerosene, and they test-run their turbine every 6 months to make sure it can carry the load. CO's just don't go down, the uptime of switches is measured in decades, and most offices have never had a power hit on the equipment side.

It's possible that your line doesn't come straight from the CO, rather it hops through a remote equipment location (a "hut" or "vault"), which has batteries but no generator.

Huts, vaults, and c

Quebec Side

[MichaelMarch]

I live here in Quebec and we never lost any power. However my ISP never came back online until 24hours later. I was very disapointed with the results, however I really don't think it was the ISP fault more to the fact that Ma'Bell(www.bell.ca) had not backup power other then a 10min UPS running there routers.

Best Catastrophic Event of August 2003

[kolombangara]

I love the fact marketers and politicians didn't get in on the pre-attack capitalization & exploitation by preplanning it in advance like they usually do these type of "nationwide" events. The Blackout probably saved Americans 2 billion dollars in wasted energy simply by turning off the switch for 5 minutes and giving the sonic bomblastment of current draw back to God, like peace, man. So, this is currently my favorite major catastrophic event this month, because it didn't involve the DOD, CIA, FBI, ADL

MCI

[bobbozzo]

I'm in southern California, and around 18:00 PDT Monday, our T1 connection to MCI pretty much quit working to 95% of the internet. MCI said there were routing problems due to the NE blackouts... whatever.

Not exactly what you wanted to know

[Zaffle]

But a funny story none-the-less...

Back during the great auckland power crises [wired.com] of 97, my ISP was Binary Brothers, a now extinct ISP. They were a great ISP, run by a few guys who knew their stuff.

Turns out the owners were physically located on the coremandle peninsula (about 4 hrs drive from Auckland), while their servers/modem racks, etc were located in the heart of auckland CBD.

The power in auckland blinked out, and as did my net connection (I was located outside the blackout area). I rang 'em up and

human error

[ajayrockrock]

I'm on the west coast so we were suffering from the california blackouts which we went through without too many problems.

But today at our colo facility which has battery backup and generators and can last for quite sometime without city power went down this afternoon. Why? An electrician turned off our breaker and shutdown our entire rack.

never underestimate human error.

--ajay

Verizon of Manhattan was fine

[bjb]

With the rest of the city dark as the night sky, I was surprised that the land lines DID work. The cell phones worked as well, but the land lines were a surprise.
As a result, I could have used my laptop for some late night entertainment, but I decided to use my lanterns and read a book instead.

What was really interesting was that you could see the stars in the sky for the first time probably since 1977's blackout. Oh sure, you can see a few, but I mean see a LOT of them.

all good here in SE Ontario

[Sherloqq]

Home:
ISP: Cogeco Cable: No interruptions
Power: NiagaraMohawk: No interruptions (I kid you not. My neighborhood hasn't lost power for a second during the outage).
Phone (both landline and cellphone): No interruptions

Work:
ISP: RoncoNet: No interruptions
Power: unsure (prolly NiMo + generators): No interruptions

Dialup: D-. Nextel: F. SBC: A

[Myself]

For the first few minutes things were fine, my ISP was still up when I decided to shut down early, rather than running the UPS battery down hard. Cell phones were fine too, for a couple hours.

Later Thursday night I tried to get back online from my laptop, and my local ISP (who uses Megapop for dialin ports) would answer and connect, I'd get an IP address, but my packets went nowhere. Tracert stopped at hop 1.

So, knowing that the phone system was up, I simply dialed out of the affected area. Calling a POP