Redundant Internet Access? 78
Supp0rtLinux asks: "In order to meet uptime requirements and SLAs, we decided to get redundant T1's with BGP. We already had two Cisco 7200 routers and a T1. After the ISP turned up the additional circuit and we tested everything on our end, all seemed fine. But when the CO lost power and the generator failed, we had no access for 16+ hours. This prompted some investigations which revealed that yes, we did in fact have a redundant T1 with BGP setup and local redundant routers with separate UPS... on our side. However, on their side both our feeds were plugged into the *same* switch which was on the same PDU which happened to be in the same CO and was on the same sonet. And they were charging us for redundancy! Six month later, we have a truly redundant BGP setup. Each feed goes to separate CO's with the primary to the local one. This makes for separate physical switches, separate power, and we have confirmed we're on physically separate sonets. Now, the only true single point of failure is the physical cabling in the street, but in CA that doesn't get damaged very often. To those of you on Slashdot who know what I'm talking about: are your circuits truly redundant? What have your experiences in network redundancy been? How have you gotten past the sales guy to a tech that knows what redundancy really means? Have you been able to prove your redundancy? Have you found yourself paying for something that you weren't really getting?"
Not there yet (Score:5, Informative)
Having them in two COs, redundant everything, yet linked to the same AS(when it isn't mine) makes me nervous.
Re:Not there yet (Score:5, Interesting)
Re:Not there yet (Score:2)
The problem is the ILEC. (Score:5, Informative)
You might get a line from Sprint that goes through Chicago, and another from MCI that comes from Dallas, but when they get to your town, they hand it off to the ILEC, who runs the last mile.
Even if it was hooked up to a different switch, or was terminaed at a different CO, you still have redundancy problems -- odds are, the lines come into your building at a fixed point, which could be hit by a backhoe.
I know of an ISP that was serviced directly by a CLEC (the city-run cable company pulled fibre to them, besides the copper run from the ILEC...) but they were run on the same poles, so it didn't matter.
The only really redundant systems I know of didn't use wires for one of the components. Typically, they had lines pulled to two different places, through two different COs (in once case, in bordering states, that were on different power grids), and then connected the two with microwave. This way, the second leg completely avoided the ILEC.
It's not cheap, but well, redundancy doesn't tend to be.
In the long run, you have to look at what the costs are going to be, and what sort of losses it's going to prevent, and if the additional benefits are going to outweigh the cost.
Oh -- and typically, even if a CLEC (competitive local exchange carrier) has their own switch, the last mile is still typically handled through the ILEC, which puts you back in the same boat. Even with DSL, it doesn't matter if there are two different DSLAMs, if they're routed through the same CO or SLIC.
Re:The problem is the ILEC. (Score:2)
I guess I should have written a two page article about things that weren't my point, so people who wanted to ignore the routing aspect of my post didn't have the excuse.
Re:The problem is the ILEC. (Score:2)
Re:Not there yet (Score:2)
Connection #1 is a cable modem with the cabling running in a conduit on the outside of the apartment building. Connection #2 is a regular modem with the wiring running through the building interior. Two separate ISPs, with separate POPs. The only common failure point is the mainboard of the computer functioning as a router -- absolutely everything else is redundant.
Admittedly, the switchover time isn't impressive, and the backup connection is slow, but I've never been without i
Actual conversation (Score:5, Funny)
Boss: We need redundant connectivity and power.
Sales-Goof: You can have as many people open browsers on as many computers as you want.
For comparison and not a plug, when my boss asked the IBM guy, he pulled out charts and wiring diagrams to explain what they had.
Re:Actual conversation (Score:2)
Did they have BROWSERS back then???
Do not take it seriously, just my attempt at a light-hearted joke!
Paul B.
Re:Actual conversation (Score:2)
On a Wing and a Prayer (Score:5, Funny)
I have two homing pigeons [faqs.org].
If Cupid smiles on them, soon I'll have even more redundancy.
Re:On a Wing and a Prayer (Score:2, Funny)
Great, now I will mod you up +1 Redundant.
Now where is it? Aha...there...
Very concerned (Score:5, Interesting)
My issues from there:
1. How do you convince an ISP to bring a feed in from another CO? Distance is a huge problem--they don't want to run it.
2. How do you know what the ISP has on their end, UPS's, generators, etc? Should that be part of the SLA? Or should you demand a tour of their facilities to see where your wire goes?
3. How can you coordinate two seperate ISP's for automatic redundancy? I suppose with a LinkProof box or something. And how do you know they aren't coming through the same telco CO?
4. Should you pay to have them manage the lines and router configurations in a 24/7 scenario? Or does it work well enough to have them do the initial install and then let it run?
5. Finally, what's a resonable cost for this redundancy?
I have some more projects that will be requiring this type of setup. Am interested to hear any opinions and recomendations from experience from fellow slashdotters......
Thanks much!
-m
Re:Very concerned (Score:5, Insightful)
Unless you're in a downtown area or a tech park, forget about redundancy.
IMHO, anything facing the public that needs redundancy belongs in a colo.
Re:Very concerned (Score:1)
If you can, pressure them. If they can't do it I can't pay them for it and I will find some one else who can.(Don't let them know even if you can't) Or I'd let them know you have found a solution off site that meets your needs, you will no longer need their lines unless they can provide what your paying for.
Re:Very concerned (Score:2)
Google for BGP.
You need to get an IP address block that both ISPs are willing to advertise/route for you. This is not a problem if you deal directly with Sprintlink, UUNet/MCI, AT&T, or another Tier 1 provider. Any modern mid-range Cisco (or Juniper) router can handle multiple connections via BGP. The main limitation is RA
Longer term! (Score:1)
2 to the same provider is not redundant (Score:5, Informative)
-Randy
Re:2 to the same provider is not redundant (Score:1)
The redundancy is handled by the 1750. We have 5 static IPs with the DSL and 1 static with Cable. Since the DSL is the primary connection, we have the routes for it listed before the route for the Cable. The momen
Windows (Score:2, Funny)
Aw! You're making this too easy.
One Example (Score:3, Interesting)
Part of the expense was justified by cost savings using VOIP between the stations and the operations centers.
Re:One Example (Score:1)
Another completely different approach (Score:5, Insightful)
The idea here is to think of your internet connectivity as two different classes of services. You should place your internet reachable servers in a good co-lo. Get BGP lines from two different sources and multi-home the boxes. Don't run your own AS (use the upstreams space) but instead place your servers "close" to your provider's edge routers. In the end, you are BPGing the loop and it is hard for 100ft of cat-5 to fail. In the end, you have to ask yourself "Am I more qualified to keep my BPG up than is Level-3 (or Savvis
In terms of your office, stick to client-only type services. Get two "diverse" connections. This might be a T-1 and a DSL, or a DSL and a cable modem. By using completely different architechures, you can get incredible diversity without spending a bunch of money. You can then IPSEC your local net over the client-only connection back to your addresses in the co-lo and with the help of a little client-side monitoring, auto-switch when a line goes down.
We offer something similar as a part of our hosting offering for users with green-screen (telnet, serial terminal) applications. A client gateway application manages logical "connections" back to our multi-homed central servers walking around BPG router "flaps" and other transient outages that BGP does not even address.
Re:Another completely different approach (Score:3, Interesting)
If anything, my "real" motive is to keep people from putting servers in-house. If your office has the same "pipe", "power", and "security" as a good co-lo, then you spent too much money building it.
After all, there are millions of square feet of unused co-lo at rock-bottom prices just begging for more space-heaters (er
Re:Another completely different approach (Score:2, Informative)
This is not just due to stability/reliability concerns but mostly security; how would you feel if your banking account was housed on www.cheap-ass-pr0n-servers.biz or something like that?
Don't be fooled; any techy at a CoLo can look at your data if (s)he wants to.
Re:Another completely different approach (Score:3, Informative)
Many banks do run in Co-los. We have neighbors in the co-los we are in that are banks, insurance companies, medical, etc. And I would feel very comfortable with my bank locating in some of the co-los that we are in.
Case in point are co-los with "real" security. Savvis (formerly Exodus) in Los Angeles (actually El Segundo for those that care) has armed guards, card key access, hand scanners, more security cameras than you can count, and man traps. If you need more, you can get private cag
Re:Another completely different approach (Score:1)
Re:Another completely different approach (Score:2)
Re:Another completely different approach (Score:3, Informative)
I used to be a network engineer at a large co-lo company which was acquired by Cable and Wireless after going through Chapter 11.
The data center in which I worked had a different take on man traps. They looked very much like a Star Trek transporter, and like the transporters, were temperamental and at least one of them was frequently out of order entirely. This was bad because they were made by an Italian company and every time one of them broke, a service tech would have to fly out *
Re:Another completely different approach (Score:1)
In the Netherlands, where I work, financial institutions are simply not allowed to do so by regulation.
Even so, I would be very sceptical of a colo which claims such security, they may very well have such levels but what is the guarentee, who monitors them and who's responsibility is it if something goes wrong? More importantly, will these co-lo's indemnify the company financially?
Most likely the systems in the colo wer
Re:Another completely different approach (Score:3, Interesting)
There's a friend of mine who developed a database system for some schools to run through their website. He is also providing hosting for them at his house.(after pricing out several hosting services and finding them a waste) The school system bought the server, and he got redundancy by hooking up DSL as pri
Redundant should be 100% (Score:1)
If you don't have a CLEC or ISP - then turn to DSS sat.
For those of you who have a T1 and want a cheap backup - think about ISDN, DSL, or even a Cable internet account - it doesn't have to equal a T1 but would do in a pinch for routing mail and basic traffic.
If your boss doesn't think your company needs a redundant line - go unplug the csu/dsu for an hour and
Re:Redundant should be 100% (Score:2, Interesting)
Actually you may not have to go that far. For redundancy around here we can go to the power company. Our local power company [mnpower.com] has a networking affiliate [enventistelecom.com] with their own completely separate network.
Good luck trying to get the local phone company to admit of such a situation exists in your area though.
Re:Redundant should be 100% (Score:1)
Re:Redundant should be 100% (Score:1)
Having redundant T1's from different providers is always going to be the best option, but may be more expensive that many are willing to pay for.
Diverse routing (Score:4, Informative)
-psy
single point of failure (Score:2, Informative)
mean you have redudant internet access. BGP can still fail.
That is
MCI/worldcom says it happened because a fiber was cut in ohio, which exposed a weakpoint, eventually afte
Comment removed (Score:4, Informative)
Re:wait (Score:2)
Re: (Score:2)
Re:wait (Score:2)
Re:wait (Score:1)
austad is correct in everything he said. I don't see any reason to use BGP with only one provider. It would be much better to bond the lines. But to solve the problem where you only have one CO, you probably need to talk to the telco(s) and see if they can provide redundancy at that level. If you do us
Re:wait (Score:2)
Both your T-1's go to the same ISP? Why are you running BGP then?
Umm -- maybe they go to different locations? The old ISP I used to work for had two T1s with Sprintlink -- one going to the NYC POP and one going to the Pennsakauen NJ POP. I used BGP to configure them because Sprintlink would keep advertising our routes even if the link went down.
My whole theory behind it was we could load balance our different IPs to different T1s (and with BGP we had control over it -- no such control with static rout
The system at one of my previous place of employ (Score:4, Interesting)
We had four T1s -- two from MFS and two from Bell. Of the four T1s, two (one MFS and one Bell) went to one NSP in Santa Clara, and the other two went to a different vendor in Oakland.
We even had physical plant diversity -- the Bell loops came from cable that ran along Stevens Creek Blvd, and the MFS fiber came up from the street that ran behind us. Outside of the building burning down, we were bulletproof.
Ran three years without a single minute of downtime.
My crowning glory in network design. Never again did I work for an employer who was willing to put their money where there mouth was for reliability.
Wait a second... (Score:4, Informative)
Let me get this straight... you're complaining that a PDU went down at your provider yet you're perfectly happy that you're running both circuits over the same cable under the street? In California?
Cables are cut all the time. Stupid things like rain water seeping through insulation take down entire city blocks. A single earthquake can disable hundreds of square miles for weeks or months.
On the other hand, you rarely hear of the type of failure you experienced. A well designed data center can take quite a lot of failure without a significant (or any) reduction in service level.
Maybe your provider is different, but all the data centers I've ever dealt with have multi-path redundant power routing systems. If a PDU goes out, another one takes over. They constantly share the load yet can easily take it over if one or more fail.
Add to that the standard AC-DC-AC power path and you've got a pretty rock-solid power distribution system.
Unless you can completely eliminate your single point of failure, you're going to be at risk for down-time. In fact, even with a completely redundant infrastructure, things have a bad habit of conspiring against you anyway.
Redun-what? (Score:4, Funny)
The IT at "my" company seems to love single points of failure. Their motto seems to be "if there is a way to build a SPoF, do it". Recent examples:
The "services office" (where IT, language service, human resources and so on work) is connected through a single line to the "main office" 10 km away. One day, an excavator cut that line. Result: No one could work for hours, because each and every device including all computers and all printers use DHCP to get an IP address. And the DHCP server (and the DNS server) is located in the main office. There was a dedicated print server, but it was not allowed to work as DHCP and DNS server.
All servers in a remote office run on a single UPS. One day, yet another evil excavator cut the power line. All rooms went dark, the UPS switched to battery, all servers were running smoothly. The PBX had and still has no UPS, so only mobile phones still worked. The hotline of the local power authoritiy told us it would take some hours to get the line fixed. So we needed to shut down the servers before the UPS battery was drained. But except for one or two servers, our IT supporter had no privileges to shut down the servers, so it had to be done from the main office. But neither the ethernet switches nor the router to the main office were connected to the UPS. We finally decided that the servers had had enough time to write their caches to the disks and simply disconnected them. And no, the UPS signal output was not connected to the servers. Now, it could signal a power outage and a low battery via ethernet -- if the switches were connected to a UPS.
Did I mention that all servers in that remote office are connected to a single switch (out of three), using up to three ethernet lines?
Did I mention various air conditions that can not cope with the heat of the servers on a hot summer day?
Did I mention that all remote office data lines (yes, one line per office) end in a single point in the main office?
Did I mention that we have a single mail server (MX for the domain) at our provider for all incoming external mail which is regularily blacklisted and that our internal MX consults that black lists to fight spam?
(Hmm, I should really stop here or I won't finish until tomorrow.)
Tux2000
Score +1, Sad (Score:2)
Redundant T1s working for me (Score:1)
I only have one Cisco, and I know the copper shares some of the same poles, but a month after swapping, two of my T1s went down for 10 minutes when something
After some thought... (Score:4, Informative)
If it's feeding a customer service center or a bunch of bratty executives or something, well... your fucked ;) never mind what I said.
Physical redundancy (Score:4, Funny)
When WorldCom merged with MCI, then bought the other provider, no one thought much of it. Until a trenching machine trenched across one of the big trunks
Got you beat! (Score:3, Interesting)
It is well known that even if at any given time you are making use of different sonet rings, circuits get shifted around based on demand, and you could end up being rerouted onto the same circuits without any notice. They only way to know is to wait till a problem occurs, and see if it impacts more than one connection.
The one thing missing.... (Score:4, Informative)
There are providers that have multiple backbones, from different providers. I worked for an ISP that at the time had 4 different backbone providers. While there, I saw one of the backbones fail, stay down for several days because the backbone provider dragged their feet in fixing it. Everything else kept working, though, and the only difference was that during absolute peak useage, servers were very slightly slower in responding due to the missing bandwidth.
Being redundant between you and your provider isn't enough... ask if your provider's connection is redundant as well.
SBC Served? (Score:3, Insightful)
For the record, I am not an expert on this, but I have a bit of experience under my belt.
cheapest onsite redundancy? (Score:2)
I should probably keep researching Zebra [zebra.org] and lartc [lartc.org] and stuff.
Re:cheapest onsite redundancy? (Score:2)
Re:cheapest onsite redundancy? (Score:2)
DNS based failover isn't a good option for servers because of these reasons:
Re:cheapest onsite redundancy? (Score:1)
Redundancy needs verification (Score:2, Interesting)
Real redundancy costs real money.
I work in a professional Colo facility in Denver, and we are fully redundant in all systems. Once it leaves your box, there's two of everything. Dual power to the box, dual network connections
Different mediums (Score:2)
re: Redundant Internet Access? (Score:2, Informative)
And that is why we called it redundant lines - ensuring if one fails the other would be able to keep us alive.
I never trusted any ISP about their claims on redundancy. Perhaps it's the competition in the business segment they are into, or whatever else, I've found ISP's to :
1. Ignorant about basic concepts, or at least not in sync with customers about them
2. Lying. As h
Re: Redundant Internet Access? (Score:3, Funny)
Hah! You call that redundancy? Real redundancy is when you own TWO companies doing the same thing.
TowerStream (Score:4, Interesting)
# Chicago, IL
# New York, NY # Greater Boston, MA
# Greater Providence, RI # Newport, RI
# Westerly, RI
TowerStream [towerstream.com] may be something to look into. I use them as our primary connection at the office - they are far cheaper than a traditional T1 ($350/mo for 512k, $500 for 1.5mbit, they can handle around 5GBit max I believe).
True line-of-site is not required, a reflected signal is usually sufficient. An external flat-panel antenna about 6 inches tall and wide is required, however. With ours setup on the roof, we get 0% packet loss, and have had no problems through heavy snow, rain or thunderstorms.
I have occasionally had connection issues, where the wireless modem has needed to be power-cycled. I suspect, however, this is simply due to it overheating
Joking and Seriously (Score:4, Interesting)
if you want to find out about "redundancy" find out what they do in the military.
Cost is another matter....
Yes Yes (Score:2, Funny)
Local Echo (Score:2)
True Redundancy is Best Achived... (Score:1)
Of Instance one T1 from AT&T and one from MCI.
Earthquakes? (Score:2)
I'm not sure if a T1 is copper or fiber- but I doubt very much that either could withstand a separation and dislocation of concrete.
Re:Earthquakes? (Score:1)
Re:Earthquakes? (Score:2)
we thought we were redundant (Score:1)
Backup lines (Score:2)
Odds are You Can't (Score:1, Troll)
If you get diverse paths to multiple CO's those CO's may share a common backhaul to the next more metropolitan area.
In most locations you can't get lines from anybody but the local telco, and all the lines run together.
In most locations if you can get lines from different providers they run along the same poles.
Most comp