Redundant Internet Access?

Supp0rtLinux asks: "In order to meet uptime requirements and SLAs, we decided to get redundant T1's with BGP. We already had two Cisco 7200 routers and a T1. After the ISP turned up the additional circuit and we tested everything on our end, all seemed fine. But when the CO lost power and the generator failed, we had no access for 16+ hours. This prompted some investigations which revealed that yes, we did in fact have a redundant T1 with BGP setup and local redundant routers with separate UPS... on our side. However, on their side both our feeds were plugged into the *same* switch which was on the same PDU which happened to be in the same CO and was on the same sonet. And they were charging us for redundancy! Six month later, we have a truly redundant BGP setup. Each feed goes to separate CO's with the primary to the local one. This makes for separate physical switches, separate power, and we have confirmed we're on physically separate sonets. Now, the only true single point of failure is the physical cabling in the street, but in CA that doesn't get damaged very often. To those of you on Slashdot who know what I'm talking about: are your circuits truly redundant? What have your experiences in network redundancy been? How have you gotten past the sales guy to a tech that knows what redundancy really means? Have you been able to prove your redundancy? Have you found yourself paying for something that you weren't really getting?"

Actual conversation

[BrynM]

This is what I overheard when the place I worked at years ago was shopping disaster recovery sites. Mind you, this was for a mainframe - this place was supposed to be fully redundant in about 20 other ways as well.

Boss: We need redundant connectivity and power.
Sales-Goof: You can have as many people open browsers on as many computers as you want.

For comparison and not a plug, when my boss asked the IBM guy, he pulled out charts and wiring diagrams to explain what they had.

On a Wing and a Prayer

[orthogonal]

To those of you on Slashdot who know what I'm talking about: are your circuits truly redundant? What have your experiences in network redundancy been?

I have two homing pigeons [faqs.org].

If Cupid smiles on them, soon I'll have even more redundancy.

Windows

[Anonymous Coward]

"Have you found yourself paying for something that you weren't really getting?"

Aw! You're making this too easy.

Re:On a Wing and a Prayer

[Anonymous Coward]

Great, now I will mod you up +1 Redundant.

Now where is it? Aha...there...

Redun-what?

[Tux2000]

The IT at "my" company seems to love single points of failure. Their motto seems to be "if there is a way to build a SPoF, do it". Recent examples:

The "services office" (where IT, language service, human resources and so on work) is connected through a single line to the "main office" 10 km away. One day, an excavator cut that line. Result: No one could work for hours, because each and every device including all computers and all printers use DHCP to get an IP address. And the DHCP server (and the DNS server) is located in the main office. There was a dedicated print server, but it was not allowed to work as DHCP and DNS server.

All servers in a remote office run on a single UPS. One day, yet another evil excavator cut the power line. All rooms went dark, the UPS switched to battery, all servers were running smoothly. The PBX had and still has no UPS, so only mobile phones still worked. The hotline of the local power authoritiy told us it would take some hours to get the line fixed. So we needed to shut down the servers before the UPS battery was drained. But except for one or two servers, our IT supporter had no privileges to shut down the servers, so it had to be done from the main office. But neither the ethernet switches nor the router to the main office were connected to the UPS. We finally decided that the servers had had enough time to write their caches to the disks and simply disconnected them. And no, the UPS signal output was not connected to the servers. Now, it could signal a power outage and a low battery via ethernet -- if the switches were connected to a UPS.

Did I mention that all servers in that remote office are connected to a single switch (out of three), using up to three ethernet lines?

Did I mention various air conditions that can not cope with the heat of the servers on a hot summer day?

Did I mention that all remote office data lines (yes, one line per office) end in a single point in the main office?

Did I mention that we have a single mail server (MX for the domain) at our provider for all incoming external mail which is regularily blacklisted and that our internal MX consults that black lists to fight spam?

(Hmm, I should really stop here or I won't finish until tomorrow.)

Tux2000

Physical redundancy

[crmartin]

I did a bunch of Wall Street work some years ago; we had an experience with this. The system was set up with two high-bandwidth redundant paths, leased from two big providers. (MCI and someone else, I don't remember.)

When WorldCom merged with MCI, then bought the other provider, no one thought much of it. Until a trenching machine trenched across one of the big trunks ... and we found out that the physically redundant lines had been consolidated into the same trunk.

Re: Redundant Internet Access?

[TheLink]

"except that in the company I used to own, we had two *separate* T1 lines from two *separate* ISP's"

Hah! You call that redundancy? Real redundancy is when you own TWO companies doing the same thing. :)

Yes Yes

[Hard_Code]

of of course course my my circuits circuits are are redundant redundant