Routing to Multiple Providers with Linux?

Dale Woolridge asks: "Suppose I want to increase the availability/reliability of my home network's connection (to the internet) and high-speed & low-latency are important. I've been using cable for almost two years now, but there are sometimes inexplicable outages which leave me without net access for hours at a time. Rather than switch to another type (aDSL) of provider, I augmented my connectivity by adding another provider. I want to automate the selection of outgoing route, but am willing to live with manual failover in cases where a particular interface/route is available but my upstream provider is having routing problems. Basically, I want to be multi-homed, but my choice of options is limited; clearly, I cannot fiddle with upstream routing."

"To that end, I did a little searching and came up with a few items to help me along the way. First, I had to download the iproute2 package, add a routing table for multipath routing based on this example, and do some interface/packet matching using this information. Unfortunately, none of this seems sufficient to do what I want. Based on what I've read, the issue lies in source address selection. If the multipath table is my default table, I think it chooses the source address based on the weights of the hops in that route. Even with interface selection rules (ip rule add dev eth0 table ...) the issue (routing problems to/from linux box) persists. While I am using the 2.2.19 kernel, I would consider moving to a newer kernel iff I can achieve my goal with it.

Based on the above, my question is this: Has anyone successfully configured their home network as described above under linux? I am interested in knowing how to do it."

Re:BGP

[Indomitus]

If you can get a cable provider to peer with you using BGP, I'll dance a jig. There are so many issues involved that even SDSL (usually a business service) providers will very rarely run BGP with you. It's the best option for your situation IMO but like I said, I'll be very surprised if you can get any cable ISP (let alone 2) to peer with you.

Re:God, are you are so geeky its scary.

[ptomblin]

I, too, have RoadRunner and it rarely goes down for more than a few hours at a time, and that's pretty rare for even that (I once had a 184 day uptime on my router box, and I usually reboot that when the RoadRunner connection drops just to make sure it's not my fault). However I know of other people who lose their high bandwidth connections for days at a time on a fairly regular basis. Since I'm running a couple of web sites and mailing lists off my server at home, if I were in that boat I'd probably look into some sort of redundancy too.

--

You missed...

[TBone]

...the "iff" used in a gramatically correct manner, too.

No one should ever use "iff" in speech unless you're talking about flying fighter planes.

This space for rent. Call 1-800-STEAK4U

You could always use NAT.

[don.g]

IIRC the Linux 2.4 NAT implementation will allow you to loadbalance your outgoing connections over multiple IPs. Your ISPs may have implemented filtering so only IPs within their netblock can get out, so you'd need to set up routing so that packets with the source address of one interface didn't leave via the other.

If you can make the interfaces go down or the NAT rules change or something when the connection drops, you've got (semi-) reliable internet access.

--

Multihomed Consumer Internet

[OnyxRaven]

Bah, here I was just wondering if there was a way to get true multihomed consumer type internet - through a cable line and a dsl line. I'm moving in next door with my buddy and one of us was going to get cable, the other adsl, and we were going to share eachothers connections to get multihomed... but from the looks of it, its just not possible.

Is everyone sure that this is the case???

Re:BGP

[Gravatite]

While BGP would surly be a good solution to the problem, it's highly unlikly that consumer grade broadband providers could be convinced to give their customer a BGP feed.

On top of that problem, a majority of these types of providers use non-portable IP space which would make the BGP feed useless anyway.

Re:BGP

[DeathBunny]

Fat chance of that. To use BGP you'll need your own block of IP's. *Not* IP's your ISP assigned, but an actual IP block you own. These are pretty damn hard to get a hold of these days. You'll also need to get a BGP AS (Autonomous System) number. Your unlikely to get an AS number assigned to a private individual or even small company.

BGP

[galmeida]

You'll have to use BGP. Maybe gated, see http://www.gated.org/ [gated.org]

Look at other options...

[stienman]

If you're willing to pay $80/mo or more for a reliable connection, you might want to look at your other options which have a guranteed uptime and latency. SDSL can get you both low latency, high speed, and most companies offering it have good gurantees (try Bullseye [bullseyetelecom.com], for instance).

You could also co-locate an external machine with a static IP, and form a vpn with it with your two links. Perhaps you could even find a host that could set that up for you for less than the cost of co-location. I know, bad latency, but do you really expect to have all three: Fast connection, low latency, low cost?

Part of the issue is that you're trying to take two consumer grade connections and create one business-grade connection. This isn't a 1+1=2 situation at all, you'll be lucky if 1+1=1.25 in your case, and if it is what you choose to do then you might as well use it for automatic failover with the associated glitch of losing any current tcp sessions when it switches. There is just no easy way to get around that with two dynamic IP connections and nothing else. If you can get your cable company(s) to give you static IPs, you might be able to finagle a few things, but you will not be able to mess with the many routers that you'd have to reconfigure to have completely transparent failover.

Often you will get what you pay for in this business, and what you are asking for is going to cost you either:
1) Try to make two broken connections equal one good connection: Work with your software and service providers and get things going (40+ hours, and ongoing problems, say 2-4hours/mo, as well as your connection fees of $80/mo or so) or
2) Skip the pain and simply do it right the first time: Get a dedicated business class connection with gurantees for $120/mo or more depending on your connection speed. A T1 (fractional, burstable, there are lots of options that reduce your cost) is surprisingly affordable these days, if you want better latency and reliability.

But then, your time and effort may be worth less than $10 an hour to you.

-Adam

This sig 80% recycled bits, 20% post user.

Re:Failover rather than multihoming

[Lurch00]

I was looking into this a while ago in the context of sharing two connections to increase bandwidth and reliability. You won't be able to get upstream routing, so most people doing this sort of thing just use some sort of failover script.

In addition, you can configure the routing table to basically use round robin on the two lines, so one connection will go out over one line, and the next over another. Note that this is at the connection/session level, not the packet level. So once you start a download (or upload) those packets will always come in (or go out) over the same interface. Your load balance won't be as good as packet by packet routing, but you should still see increased throughput. Might as well make use of the bandwidth, if you've got it..

Try this....

[NetJunkie]

Take a look at the NAT routers from http://www.nexland.com/. They have one that will load balance across two connections. Just plug in two Ethernet cables and away you go. Since you have two addresses that are not portable, it will "only" load balance a connection, not by the packet. But it should work!

Re:Failover rather than multihoming

[Asgard]

What about round-robin'ing the outgoing *packets*, and choose just one source address for things like games. Sure, all the packets come back one way, but most of these providers have decent download, just bad upload. You could speed up web-surfing by using squid to round-robin the source address.

Small LAN solution

[PGillingwater]

For my company, we wanted redundancy, but felt that it was a waste to have two providers with one unused. Therefore, I set up two firewalls. One of them is the default gateway for our LAN, and the other runs a Squid proxy. All our Web surfing through Squid goes via cable, while the other traffic goes via DSL. If the DSL fails, we can switch the default gateway with DHCP (just use a low lease time, or do it manually), and if the cable fails, (as it does often) we just switch off the proxy and surf direct. Works fine for us with NAT and IP chains.
--
Paul Gillingwater

Re:Not a chance (OK, very slim chance)

[penrose]

Actually, they won't list anything smaller than a /19 which is 8192 addresses.

Re:BGP

[StandardDeviant]

You don't want BGP.

But if you did, try GNU Zebra as a nice alternative for all your heavy-duty rouing needs. (GNU Zebra homepage: http://www.zebra.org, site seems to be down at the moment)

--
News for geeks in Austin: www.geekaustin.org [geekaustin.org]

Failover rather than multihoming

[elbuddha]

You have 2 basic problems with this scheme:

1. You have been provided with 2 different IP addresses or address spaces, one from each provider, neither of which is portable (neither provider is going to accept traffic from you with a source address in the other provider's IP block).
2. Neither provider is going to announce routes to you, period. Not with BGP, not even with RIP. So forget about doing path selection with any routing protocol

Sounds like what you really need is a failover/shadow scheme rather than actual multihoming.

The way I would go about that goal is:

• Figure out what manual commands you go through in order to switch over your connectivity from one provider to the other, and back. Write those commands out in a script of your choice.
• Determine a mechanism and write another script for determining if a connection has gone bad, and run that script as a cron job or background task. You may even desire to combine the two scripts, but personally I would keep their functionality separate.
• Have the connection-testing script call the connection-changing script when your connection goes down. You could even have the testing script then test the new connection as change back to the first if it goes down, or continue to monitor the primary connection and switch back when it comes up.

More work for you of course, but have fun with it. Do a good enough job and someone else may even want to use it too.

Re:Not a chance (OK, very slim chance)

[autocracy]

Depends on what system you're on. Each Tier-1 ISP has it's own policy. Verio goes by /32 I believe. About a year ago, it wasn't uncommon for many ISPs to do routes smaller than /32 - but the routing tables are really hurting...

So you're a karma whore, eh? For the right price, I'll be a karma pimp...

Not a chance (OK, very slim chance)

[autocracy]

You'll need to register for an autonomous system (AS) number, get yourself into the listings on the major routing tables (warning: nobody wants to add a /32 listing - better get 255 IP addressess for each connection!), and you'll probably have to get your ISPs to play nice too. Oh, and I doubt they'd change their route tables to provide you with a new block of IP addies.

I know that this how things sounds harsh and cruel, but the truth of it is that it's probably not going to happen the way that you want.

So you're a karma whore, eh? For the right price, I'll be a karma pimp...

God, are you are so geeky its scary.

[Beowulf_Boy]

"My Cable ISP goes out once a month, so I bought another Cable connection for 40$ a month so I will never be without my connection"
You are sad, very sad.
I have RoadRunner, it only goes out when the electric does, so I can't use it anyways.
Oops, I forgot, you have your basement filled with leadAcid batterys so you will never be without a computer.
Lemme guess, you also have an I-opener in the Crapper so you can check your e-mail when "Other" business arises.
Go outside, get some fresh air, take a walk, something, but don't pay 40$ a month so you will have a backup connection for a few hours a month or so. Thats redicoulas.
IF you really want to, I would use Freesco, it allows connecting 3 networks together, and it just sees the cable modem as a network, I use it right now, ( I have it running on a p75 with 4 megs Ram) and it works great, Runs Linux, and was a sinch to install.

DynDns and a dial up..

[jimmydigital]

I had kicked this idea around for a while... but have never gone forward with it because my cable connection is so stable.. this was my idea though.

Get a cheap dial up 56k service or dsl or whatever for your budget.. then you should be able to write some simple scripts to bring up the backup connection when the primary fails.. change your outgoing routes.. and for incoming connections use the DynDns [dyndns.org] service to handle your domain(s). They have a service that allows them to host your domain and tie the IP into their dynamic dns service...

freesco

[jimiUK]

use an old 486 as a router running freesco ( www.freesco.org ), use three nic's and check the freesco forums for scripts to automatically change routing if your primary route goes down . check my freesco box at http://www.uptimes.net/hostinfo.html?popup=1&hid=1 2215