Affordably Aggregating ISP Connections? 180
An anonymous reader writes "Has anyone setup a system to aggregate multiple ISP connections to form a high bandwidth site-to-site link? Load Sharing SCTP looked interesting, but it doesn't look like it has been widely adopted. Multi-Link PPP appears to be more widely supported for clients, but I can't find any good guides for setting up both sides of the connection for a site-to-site link. The hardware solutions I've found are expensive for a small business. Does anyone have experience using hardware solutions from Mushroom Networks (Virtual Leased Line, p2 of this document), Ecessa (site-to-Site Channel Bonding), or others?"
pfSense (Score:3, Interesting)
Ask Willie Nelson, he uses Mushroom Portabella (Score:4, Interesting)
Mac OS X? (Score:5, Interesting)
Admittedly, I have no idea if it works, nor do I have any idea how it decides to load balance between the connections.. But I ran across the feature the other day and it looked pretty cool.
In Mac OS X you can create a new "Aggregate" network device from any other devices and, in theory, do exactly what your describing. Again, I just ran across this the other day in Network Preferences and have no idea if/how it works, but it might be worth a shot (especially since it seems a lot easier to configure than a roll your own router with dd-wrt or tomato, though those likely offer more fine-tuned configuration).
LISP Routing (Score:4, Interesting)
Some people (Cisco, etc.) are working on developing the Locator/ID Separation Protocol [google.com] as a core component of the Internet infrastructure.
If that ever takes off, you'll be able to buy a Provider Independent IP address block, advertise it through multiple ISPs (even Cable/DSL), and transparently load balance your upstream and downstream traffic across them, without bloating the core BGP tables.
The downside is, you'll have to use an MTU that's smaller than 1500, but I'd say it's a fair trade.
Re:Peering (Score:2, Interesting)
I know linksys has a couple routers (both the RV042 and RV082) that supports 2 incoming broadband connections with link aggregation (or it can use it as failover) if you used two of these and set up a VPN it would be fairly cheap/easy (under $500 easy) I just looked on their site but since the Linksys business stuff is now buried in Cisco's crappy site, i was unable to find a link. I've seen them at Fry's plenty of times. I've used several of them and they tend to be fairly stable.
I looked into the RV082 a while ago and found that you can get reasonably close to doubling your _outbound_ bandwidth, but not inbound. Bonding the inbound links would require both WAN lines be provided by the same ISP, so they could configure round robin across your two links.
The RV082 is a great little SOHO router and does pretty good load balancing/aggregation of outbound traffic. The OP seems to be looking for true bi-directional link aggregation of dissimilar ISP WAN links (cable/dsl or two of one of these from different providers). This is simply not possible, because there must be intelligence on the other end of your links round robin'ing the traffic between them, just like your RV082 is doing in this case.
In short, this is a great inexpensive product to double your outbound and provide redundancy. Keep in mind you'll need to do some creative things in DNS and with port forwarding on the Linky as you'll have two different public IPs on those WAN links. WRT hosting a mail server, you'll need two MX and A records, one for each public IP on each WAN link. You'll also need duplicate records for all your servers, whether WWW, ftp, etc.
Setting up _inbound_ redundancy is not simply clicking a radio button as with outbound redundancy. Remote hosts have to be told how to reach you. This means advertising both routes. Since you aren't paying an ISP for this redundancy, and you're doing it on the cheap yourself, you'll have to mangle DNS to get the inbound redundancy.
If you're looking for merely link aggregated high bandwidth site-to-site, I'm not sure if this Linky will do so with the VPN feature. You can sure try it. You can also use the little brother RV042 for a little less money, although neither is terribly expensive.
Re:Pering (Score:3, Interesting)
Its long, at least read about Greenlight in N.C. and learn!
I am 100% positive you could do this with hardware that will run the DD-WRT, here is a list of DD-WRT supported devices [dd-wrt.com], they have a search link, but I find that it does not work very well if you do not know the name of the router / firewall that you are looking for. So use the list and find a supported device.
You would need two of them and two different providers. You could even get a third one and do some special VLAN stuff to put some ports on all three on the same virtual network., many options.
These devices are very light weight, therefore shipping is next to nothing. The Linksys WRT54Gs' were great routers for the DD-WRT software. Costing over $75 when they first came out, dropping to $69 for years and finally hitting $15 or $30 when the stores were unloading them to bring in the new Linksys routers (none of which will support the DD-WRT software, except one that runs Linux). NOTE: there are BETTER routers than the WRT54G to run this software. The WRT54G will ONLY run the Micro version of the software. Do yourself a favor and get one that will run the Mega version of the software! (They cost less than $100 per and well worth the price.)
Linksys (Cisco) begin removing DD-WRT compatible firewall/routers from store shelves, replacing them with devices that are NOT compatible with the DD-WRT software in 2007/2008.
Get two DSL lines ($13 - $19 each), add in a NAT and a couple of these routers, probably need to do some secure tunneling to avoid the DNS of the Cable / DSL Companies and voila you are good to go. Your DSL speed will vary based on distance, but even far away you can get 1.5MB down and 384Kbps up. If closer you can get 3Mb down and 768Kbps up. (That is faster than 98% of Americans with Cable Modems because of throttling of service by Cable providers.)
Could you run the second DSL upstream over the first one? Thus saving the cost of a second telephone line, you would lose the redundancy that two telephones would provide, but save around $13 per month on a second phone line...probably better just to get the two lines, you total cost of ownership (TCO) will still be less than $60 per month and you will have redundancy. If one service gets stupid and starts throttling, drop them and get a different one. Politicians help us if they all throttle!
Solves allot of problems related to Cable companies throttling back service if you can create a secure VPN that their Deep Packet Inspection and/or Bandwidth shaping (throttling) service might have a harder time restricting (throttling). Granted they would still throttle you back by your IP address or MAC address of Cable Modem. Again, they do that now anyway.
A friend of mine was pissed that he was throttled back to less than 100K down and 0K up 85 - 95% of the time. He went on and paid his cable company the $10 burst / protection racket money / "give me a little more of what I am already paying for money" extra fee. Keep in mind that they were promising up to 8MP and delivering less from day one. He said he got a letter in the mail that they would be rolling out a new service in his area, the day after they started using that service, his bandwidth was throttled to next to nothing. (0 Kbps upstream, consistently less than 20Kbps). (There were 1 GB, 2GB and 3 GB ~ 1 second spikes ONLY, unless he was downloading a Linux distro, then he got 3GB - 4GB sustained with a 1 sec 6GB spike) He is convinced that they throttle him back because he uses Skype VoIP service (uses P2P packets) in a vain attempt to get him to switch to the Cable companies VoIP service. At less than $100 per year, Skype blows away any telco/Cable company offering.
Guess what his speed was after the switch over....Yep less than 100K (down) and 40K upstream 95% of the time. When he is throttled back to 0Kbps like I am, t
Packet reordering / VPN Bonding (Score:2, Interesting)
Obviously direct aggregation isn't possible, as each line will have a different source IP. What works is load balancing, but load balancing sucks. If you do per-TCP-connection load balancing on multiple lines, lots of sites will give you problems, as multiple requests for the same session are coming from different IPs. Online banking doesn't like this, ads-supported sites often don't like this (as the ad was loaded from a different IP). So this leaves you with per source-host load-balancing, and this only makes sense if there are lots of people who are two share the lines.
Doing real aggregation (bonding) requires a remote endpoint obviously, located in a datacenter somewhere for example. Problem: There is no standard protocol that works for a combination of different lines, Multilink-PPP will only work for several identical lines from the same ISP (ideally using the same clock source at the DSLAM etc). Why is that? That's because if you use multiple lines, they will have different latencies / round trip times. And if you bundle those, this means that TCP packets will overtake each other in-flight. So in the end whoever is receiving the re-assembled stream will get it out of order. And TCP can not differ between reordered and lost packet - if an unexpected (too high sequence number) packet is received, it is dropped. And this can not be solved by buffering at the router/PPP-device, because this buffering would interference with TCP windowing. In the end most of your aggregated bandwidth will therefore be eaten by retransmissions.
So, people may tell you to try this and that, but in the end everyone who has ever REALLY tried it himself will tell you: Forget about it, the performance will always be really bad (unless you have multiple identical lines).
There is a small german startup I work for which has solved the problem by creating a new bundled VPN protocol running on the way between the router in your office and the one in the datacenter, basically running a man-in-the-middle attack on TCP to get rid of the packet reordering in-flight. See http://www.viprinet.com/ [viprinet.com] for the available products and background info on how it works. Pricing starts at ~1000 USD, but obviously you'll need two boxes - probably not what you'd call "affordable". And sadly we do not yet have distributors inside the USA.