Working With 2 ISPs For Home Networking? 356
An anonymous reader writes "This is, I think, a simple question — but one which I can't get the answer to.
As a typical, but perhaps high-demand home user I would like to use 2 separate ISPs. ADSL is pretty cheap nowadays, and 2 x ADSL seems a better value than one fast one — especially in terms of reliability.
If one breaks, at least the other will work.
Using an old box as a router/firewall, how can I configure a system to use two completely separate ISPs in a sensible manner?
Ideally, I'd like the load of my browsing to be balanced, but at the minimum, I'd want some kind of 'fail-over.' If I leave torrents running over night, I'd like the router to use whichever connection doesn't block the traffic — and preferably for it to reset the errant connection.
Ideas?"
DSL+Cable (Score:5, Informative)
Linux distros (Score:5, Informative)
pfSense + two independant ISPs (Score:4, Informative)
Dual WAN router (Score:5, Informative)
what I use is a nice distro... (Score:5, Informative)
called Clarkconncect (http://www.clarkconnect.com/)
It's basically a CentOs (aka free Red Hat) wich can do multi-Wan. It has a nice web interface fir Firewall, ftp, web and mail server, shell..
No idea if it can reset errant connections, but it can do anything you can on redhat, including using two Wans simultaneously. (chek Clarkconnects forums for multi wan)
up and running within 30 minutes, mine has reached 165 days uptime (Bi-P3 GHz, 2 Go Ram, 4*500Go HDD, 3*Eth 100 (upgraded from a faithfull Compaq Deskpro 400 Mhz "server")- web, mail, and bittorrent dowvnloader (torrentflux-bart) as well as "media server" connected to the xbox with XBMC)
LARTC (Score:2, Informative)
http://lartc.org/howto/lartc.loadshare.html [lartc.org]
Check it out.
Those ISPs may not be redundant (Score:5, Informative)
Most DSL circuits, even sold by different vendors, go through the same facilities and sometimes the same equipment. For example, the local loop is usually the local telco's, no matter who your DSL vendor is. And many DSL vendors resell one of a few wholesale providers (e.g., Covad), so your data on both DSL lines could be going through the same wholesale provider's equipment/facilities. The same may be true of other technologies (e.g., fiber).
In trying to setup something similar, we finally settled on using cable for one circuit and fiber for the other. We know the cable company has its own local loop, and they assured us (FWIW) that they have their own facilities out to their upstream provider (e.g., AT&T, Sprint, etc.). Fiber would be Verizon. We would use DSL, but I'm concerned that it would end up in the same Verizon facilities.
Good luck. There are also routers that do fail-over, but I know that's not what you asked about.
HotBrick (Score:5, Informative)
Hotbrick makes a very good load-balancing soho router. They're a bit pricey but they seem to work quite well for exactly what you're describing. Take a look on ebay for their LB series.
I do have to second the suggestion of using Cable+DSL rather than DSL+DSL. Most places where there are multiple DSL providers, they're both operating from the same physical infrastructure with one reselling the service of the other. It's certainly better than one by itself, though.
2 ISPs? Single provider. (Score:5, Informative)
Honestly, I think that's not understanding how DSL works very well. In virtually all markets, there's one physical DSL provider, and a few dozen 'ISPs' which cost a little bit more to provide potentially 'unique' services on top. One monopoly for phone (and hence DSL), one monopoly for cable.
Er, the cheapest DSL is what, around $25, $30, for 256k? Double that, and you've got a price for very fast (8mbit or more) cable, including 256-512kbit upstream. Even if you have 2x256k, and the equipment to use it in a decently efficient manner, that's still some 512kbit, and two different IPs.
Only in a few situations can you use the bandwidth of both cooperatively for a single task, and the most common failure is based on when the physical link/line conditions deteriorate, in which case having two ports to the same network isn't going to make any difference at all.
Cable/DSL will provide the potential reliability you'd be looking for, I think. But, as a home user, some 98-99% (even if not 99.97%) uptime isn't good enough? For the additional cost, it's not worth the extra -average- hour per month of downtime you gain 'back'.
If your ISPs downtime is any more than that, you have every right to complain, twist their arm to fix whatever might be causing the problem.
3G failsafe (Score:1, Informative)
If you're more bothered about redundancy then extra bandwidth, and you're in a geographyically capable location it may be worth getting a router with a dual WAN (such as ADSL/3G). Vigor sell ones that support a 3G modem, such as http://www.buydraytek.com/draytek-vigor-2910g-p-55.html
These have good QoS options and also bandwidth on demand.
Failing that, as others have said you would probably be better with cable/some other medium as a backup. Generally DSL faults are more likely to be in the ATM/last mile section, where infrastructure is usually shared with ISPs.
Re:DSL+Cable (Score:5, Informative)
" SmoothFirewall 4.0 - Update 3
Download Update 3 Update 3
516 KB (528,827 bytes)
MD5: 85ac7940504a0fe7eef2b91016cf80f6
This update adds Load Balancing abilities to Advanced Firewall systems. It also corrects a problem with IP address sorting on some pages and updates the DHCP client to fix a theoretical vulnerability. Problems with PPTP and PPPoE clients have also been corrected.
Please install core Update 2 prior to installing this update.
Detail:
* Load Balancing
It is now possible to load balance outbound proxy requests and other network traffic in Advanced Firewall. Primary and secondary external connections are 'pooled' using the Firewall / connectivity and Firewall / secondary addresses pages."
smoothwall4 supports load balancing out of the box, no hassle, no mess, no fuss, but then, smoothwall is only free as in beer, but i find it works well enough.
Re:pfSense + two independant ISPs (Score:2, Informative)
Simple, not cheap (Score:2, Informative)
A dual-WAN router is the easiest way to go, but I wouldn't call it cheap. A decent dual-WAN router will cost you about twice what it would cost to build a cheap, but decent linux box.
Re:Point of failure (Score:5, Informative)
Even if they're from different providers, they're running over the same phone network (esp. since smaller providers are just resellers). A backhoe, lightning storm, or major power blackout doesn't give a sh*t that you went through two different providers.
Why bother, seriously? Why? (Score:5, Informative)
Seriously? Is your network infrastructure -that- unreliable that its actually worth *doubling* your costs for redundancy?
I have had maybe 10-15 hours of internet-only downtime in the last 8 years. Of that, maybe 4 hours affected me (ie I was awake and wanted to use the internet). I've had another 10-15 hours of power fail in the last 8 years, and even with backup power the internet was still down (routers, switches, etc in the upstream path weren't on backup power so keeping my 'modem' up isn't worth beans.
In any case, I can see a lot of situations where it would be worth another $2500 over that period to have had internet access for those couple hours.
If I were running servers (and I am), it might be worth it, but in practice its not worth the trouble. round-robin DNS just means every odd connection attempt fails if one of the links is down, and dynamic dns updates to take the downed link out of rotation would be great except most internet outages are over before dns updates are likely to propogate. So its just not effective.
If I wanted -faster- downloads, that might be worth 2 connections, but that's not what you claimed your objective was. And even then, it usually won't make a specific download faster, but will rather let you do 2 at once at full speed (in the case of a large http or download for example which only uses one connection) which may or may not be what you need. Torrents, using multiple connections, will of course benefit from the extra bandwidth capacity.
If you SERIOUSLY want redundancy, you might want to look at a router that can fail-over to dialup. That will actually stand of chance of being available during a power failure, and might not cost you extra in terms of service, since many ISPs give you some free dialup hours as part of your broadband. And the dialup infrastructure is often separate enough from the adsl/cable infrastructure that you'll be able to connect on dialup while adsl/cable is down.
Re:Point of failure (Score:2, Informative)
just as tepples wrote: you need 2 internet connections using different infrastructure for the last mile, or preferably more. DSL+Cable should be the right solution.
As for load balancing etc, you've got two options:
1. router with 2 WAN ports
2. any pc with 3 network cards + linux + googled up howto for 2 internet connections
Re:Point of failure (Score:4, Informative)
That's true, but you can only do so much to prevent outages. In the enterprise, if you want to avoid fiber-seeking backhoes, you get a failover location. That's difficult to do in a home network.
I'd say cable+DSL ( or maybe throw in something like the AT&T USB Connect 881 [att.com].
I'd see if it's possible to get FiOS in your area, too. That would give you the best speed, for sure.
Re:2 ISPs? Single provider. (Score:3, Informative)
Dual WAN Router (Score:5, Informative)
What you want is a "dual wan" router [google.com]. Which will give you two ways out, by default putting each connection between your local host and a remote host over a single WAN's route, but pool the two WANs so the less-full one gets the whole next connection.
Then you want to look into "bonding [wikipedia.org]", or whatever the router vendor calls their version of it. It usually doesn't work, because the two different WANs usually take very different routes most of the way to the remote host, and the bonding has to accommodate all the hops between on each of the two WAN routes. But sometimes it does work, especially if the routers at both ends of the routes share the same bonding technique.
But you will indeed get immediate uptime benefits. Because if one WAN gives you, say, 99.9% uptime, that's 0.1% downtime, which is still over 31,000 seconds down a year, which is still almost 9 hours. But if you can get connections over either one WAN or the other (each at 99.9%), you can get 99.9999% uptime, which is only about 32 seconds a year, which is unattainable at reasonable prices for a home user.
Multihomed routing (Score:4, Informative)
It sounds like multihomed routing is what you're looking for. there's a decent intro here:
http://www.oreillynet.com/pub/a/network/2002/08/12/multihoming.html [oreillynet.com]
here's what I've just got going (Score:2, Informative)
Recent events (FLOODS) have shown me how fragile my DSL service here is. My provider's DSL was down for the entire state for several days. So I called my local nephew-of-satin cable co and had them install a cable modem last week.
I run a web server, mailserver, and numerous other hobby services here, so I had the "business grade DSL", which is 936/1536. (divide kbps by 9 for a good guestimate in kb/sec, so 100 up, 170 down) DSL always provides me with that speed, it never fluctuates so I get every penny I pay for. I also pay a bit extra for a block of 8 (5 usable) static IP addresses which my services require.
By comparison, the cable offers many more tiers of service, and I opted for again the "business class" service. This I was told was 2k/20k. When he brought the modem I ran a speed test. The installer scoffed at those numbers (about 1.7/15k) and told me "You never really get 2/20, that's the theoretical maximum, just like DSL" at which point I had to show him what DSL really gives you.
Another entertaining surprise was that the cable co did not offer static IP addresses in my area. I talked with my "business representative" for my area of town and he agreed, "Yes that does make my job rather difficult." Offering business internet service without static IP option, I feel sorry for that salesman. Like running a grocery store but not carrying milk. My speeds were about 1.7/15k when we tested it during the install, but it's actually been clocking in very close to 2 up lately.
Not having a lot of experience in multiple simultaneous ISPs took a little digging to get things working properly. "multilink multihoming" I believe is the correct term for having two ISPs on the same machine. Being able to USE them both at the same time is the trick. Most OSs like to reply back on the default interface, regardless of which one the traffic came in on. First requirement was to get a second nic for my server. Without that, the SYN packets came in on the 2nd nic and tried leaving on the first nic, which wasn't going to work of course.
After that was settled it still didn't work, ACK packets were not being forwarded by my router. This required a special bit of software on the server, IPNetRouterX, to modify the traffic since OS X puts default gateway information on the packets even from the non-default source. (speculating this was causing the router to just toss out the packets) Ever since that it's been working very well. During my troubles I talked with numerous people and got a mix of responses. Some were wondering why I was having any problem at all, and others were telling me they fought it for a long time and never got it to work, (mostly unix ppl in both groups) so I assume some unix network stacks support this and some do not, be sure to check your distro.
Now this is with the server answering on two distinct IP addresses. This is not fail-over, it's one server that can answer requests from two different connections at the same time. Maybe not quite what you are looking for. If I wanted to use it for fail-over I would have to change my DNS entries. This would take awhile to propagate of course. But if you could update your DNS entry quickly enough, such as by getting a registrar that had a very SHORT expiration on your entries, (DYNDNS) this could work as a hot-failover. Not a matter of the backup coming online automatically when needed, but of it always being online.
A common thing to do in cases like this is to have your DNS server serve up your two (or more) IP addresses in a round-robin fashion. Try doing a DNS lookup on microsoft.com several times and you will see you are getting different IPs each time. (I currently get 207.46.197.32 and 207.46.232.182 for microsoft.com) If you have two ISPs, and hand out your two addresses round-robin, that will give you some automatic failover for your dual always-online providers, and if one of them craps out, users will just have to notice the timeout, and click the connect button a second time to connect until things get fixed.
Multihoming (Score:4, Informative)
Multihoming:
Cable/DSL
http://en.wikipedia.org/wiki/Multihoming [wikipedia.org]
Multihoming caveats:
http://en.wikipedia.org/wiki/Multihoming#Multihoming_caveats [wikipedia.org]
Get matching NIC cards.
It's actually pretty limited (Score:5, Informative)
I have quite a bit of experience with this, as I use two consumer ADSL circuits to provide very reliable 'net services at my office.
To an extent you either get to use two different services (for reliability) or combine them into one service for improved performance. Not both.
If you're going for reliability, you'll be using two different providers. That eliminates the use of multilink PPPoE to bond the two services into a single logical service with a single public IP address. It also eliminates ATM channel bonding, which is the other way to achieve the same end. This isn't such a great loss as you might think since I've *NEVER* found a provider (at least here in Australia) that knows what either is, let alone supports even one of them.
So, you're stuck with two ADSL circuits, each with separate PPPoE connections (or direct IP over ATM links; either way) and separate public IP addresses.
This sucks. You can't even load balance across them properly without the cooperation of a router/proxy on the other side of your ADSL links.
Load balancing your transmissions on a per-packet basis is obviously hopeless because any sane ISP has egress filtering based on source IP address, and even if they don't you'll still get replies back on the official source IP (so you won't gain much). SNAT won't help because if you SNAT some packets in a connection the recipient will have no idea they're part of the same connection as the unmodified packets leaving on the other connection. The only way that packet-level load balancing across multiple links with different IPs will work is if you're only talking to an endpoint (probably a VPN termination point) that is aware that you're using multiple connections and can combine them. You can use tricks like multilinked PPTP for this, or iptables trickery on each end. In any case, you're going to need access to a server with enough bandwidth to service both connections that's willing to route traffic for you. You probably don't have this.
So, packet-level load balancing is out. What's left? Connection-level, and per-protocol.
Connection level load balancing works well for some services. Outgoing SMTP, for instance, is well suited to being randomly allocated between multiple ADSL links (if you're unfortunate enough to have users who think that 100MB attachments are a good idea). Unfortunately most home user services like HTTP web browsing are not. You'll find that websites like to store session data with your IP address, so if you do connection load balancing with HTTP you'll find that websites keep on forgetting your login. To work around this you need to use "sticky" load balancing that remembers which connection was used to talk to a given host - but that, of course, reduces the benefits of the load balancing.
In the end, all you can really do is a bit of sticky connection-level load balancing when establishing new outgoing connections for some protocol types. If you want more than that, you need to do ugly things like say "all FTP connections go out ADSL1, and all SIP and other VoIP connections go out ADSL2" etc.
Personally, I don't bother even with that. I have both ADSL services listed as MXes for the company's DNS, so if one is down we still get mail. The A record points at a colocated server elsewhere on the Internet, so that's not a worry, but if it didn't I'd have to use some sort of ISP-level or colo load balancing to reroute traffic down whichever link was currently available.
Outgoing connections just all use the primary link when it's up, and fail back to the secondary link if/when the fast one is down. The secondary link is the primary MX, so when both links are up mail will tend to come in one link and everything else in the other.
If I wanted more than this, I'd probably have to route everything through another server colocated at an ISP or peering point. Unless I could get free traffic between it and both my ADSL circuits this would get expensive fast - and it'd also reduce the benefits of the redundant ADSL links
Re:DSL+Cable (Score:4, Informative)
D-Link made a (now discontinued) 4-port router that load-balanced: http://support.dlink.com/products/view.asp?productid=DI-LB604 [dlink.com]
Edimax Technology currently makes a couple of lower-priced load-balancing routers: http://www.edimax.com/en/produce_list.php?pl1_id=3&pl2_id=18 [edimax.com]
It appears that software firewall solutions (mostly linix-based) have the best support and the most features, for example: http://www.smoothwall.com/products/advancedfirewall2008/?loadbalance [smoothwall.com]
Re:Point of failure (Score:4, Informative)
I use a failover setup where the primary connection is an xDSL connection and the backup connection is cable.
Some details make failover non-trivial to do. The ADSL occasionally gets the DSL line up but no IP connectivity. The cable modem is very stable but slow. I ended up configuring linux on a small embedded computer (soekris net4801). I have a script running from crontab that pings the next-hop. If the primary connection fails, the default route is changed to the backup interface. One interesting complication is that I also use bandwith shaping with tc/htb, so iptables is configured to mark packets based on which interface they come from, which tc then can pick up and shape. I don't think there is any box/product that can fulfill all my needs, but I would have saved me much time if there were.
Re:This is what my work uses... (Score:1, Informative)
Fat Pipe is expensive, but allegedly works well. There are many, many cheap and so no cheap load balancers out there, with a diverse range of functionality. The very cheapest ($100 D-Link) just do fail over, some do load balancing to various degrees, but it's real easy to break connections for things like HTTPS if not done properly.
As for aggregating the uplinks, the options are much more limited - Fat Pipe is one, as are various also expensive Cisco solutions that rely on symmetric links.
Solutions from Mushroom Networks - http://www.broadbandbonding/ [www.broadbandbonding] - allow one sided aggregation of HTTP traffic and load balancing of the rest of arbitrary uplinks - T1, DSL, Cable etc - although still a bit pricey for home use. There's also things like ShareBand, but that superficially looks like a more hacky solution and requires some ISP cooperation.
Re:Point of failure (Score:3, Informative)
sharedband (Score:1, Informative)
You might look at a company call sharedband ( http://www.sharedband.net ). It looks like they do this exact kind of thing, bonding cable/dsl/t1/etc lines into a single pipe providing increased speed AND redundancy.
sounds like they are pretty new and i can't find too many reviews on them but they look like they may be worth a shot
Re:DSL+Cable (Score:5, Informative)
Well in my ventures into traffic shaping I've seen lots of data on load balancing as well. Most traffic shaping on GNU-Linux starts off with iptables.
A good backgrounder on iptables is at the Linux Documentation Project. (TLDP.org) You might start off with a short introduction to a simple NAT. Setting up a basic NAT is a good start before you get too far into it so you feel like you've at least got some success before you get into the deep end.
Once you feel like you've got a simple NAT down, then look for the Linux Advance Routing Howto or something of that nature. I forgot the exact title but it's close to that. That's a good one. It's dense reading, but look at the cookbook section. It has a script called wondershaper that is interesting and might give you some ideas about traffic shaping and load balancing.
Gentoo also does some nice documentation on load balancing and traffic shaping. Once you have some of the lingo down you can google around for some of the tutorials the Gentoo users have posted. Most of those tutorials can be used with the kernel that comes with Knoppix 5.0 without needing any modifications.
Personally, I think Knoppix is a good starting point for a router because it gives you a level of security in that most of your OS is read-only and the default security is pretty tight. Working with Live CDs can be a challenge if you're new to it, but a key tip is that you can quite easily modify the isolinux.cfg file on the CD to create custom boot commands burnt into a CD such as how to automatically load up your iptables scripts upon reboot. This makes a nice home-brewed embedded style device using all generic second-hand components.
Anyway, that's mostly stuff I use for traffic shaping, but it's a good start towards doing failover stuff too.
The Knoppix part may be too much of my personal preference but the part about going to TLDP and looking for the Linux Advanced Routing Howto should certainly be a good start in any case.
Here's what DSL typically shares (Score:3, Informative)
Let's start at the bottom of the OSI stack - physical layer. The wires from your house to the telco office are usually physically separate until they hit the first active device, which might be a Subscriber Loop Carrier in a big green box down the road, but is more likely to be copper all the way to the telco office. They're bundled into bigger and bigger cables (e.g. 24-pair, 50-pair, etc.) There are common-mode failures here - backhoes, wet cables, cars crashing into the telco box - but one of the most common failure modes is "technician mistakes", which usually only take out one wire pair at a time.
At the telco office, your wires get connected to a DSLAM which provides Layer 2 service (DSL is usually ATM underneath.) If both ISPs are using telco DSLAMs, then it'll probably be the same DSLAM box, but if one of your ISPs is using Covad and the other one's using telco, then you're on different DSLAMs. Some DSLAMs have integrated routers, but back when I was working more directly with this stuff there'd typically be an ATM network connecting the DSLAM to some regional concentrator network. The ATM network might have common-mode failures such as port cards, but it's mostly carrier-grade equipment with diverse physical routing.
Eventually you get to a router for Layer 3 service. If your DSL provider uses a telco DSLAM and forces you to use PPPoE, there's a good chance that you're tunneled through a telco router, but eventually you'll hit a router actually managed by your DSL provider. And from there on out to the Internet backbone, everything's basically diverse.
I don't know how Verizon does FIOS - the fiber system's obviously diverse from the copper+DSLAM system, but there might be more common infrastructure upstream or they may use different tools to concentrate it (e.g. FIOS might be using routers while DSL might be on ATM.) If you're using Verizon DSL as opposed to a third-party ISP or an ISP using Covad, you'll probably hit the same Internet peering points, so you could be susceptible to problems like "Cogent decides to have a peering fight with Verizon this time", but on the other hand your ISP might have Verizon as their upstream provider so it's a bit hard to tell. That layer's certainly much more reliable than 10 years ago.
Re:pfSense + two independant ISPs (Score:2, Informative)
m0n0wall does not support two WANs. So only pfsense qualifies.
Re:Point of failure (Score:5, Informative)
You can use more than one default route. For reliability you'll want keep some sort of connection-specific check script and reset to a dedicated route if one of the connections goes do. But while both of them are up you can use both connections together so long as you have multiple data streams:
ip route add default table "${MULTI_TABLE}" \
nexthop via "${T1_GW}" dev "${T1_DEV}" weight 1 \
nexthop via "${DSL_GW}" dev "${DSL_DEV}" weight 3
There's a bit more to it than that, but the above example is the heart of a routing policy that splits traffic 3:1 between the DSL and T1. Google should be able to show you the rest.
*nix on old hardware (Score:5, Informative)
look for the Linux Advance Routing Howto
Somewhere in that site it talks about some of the problems of having 2 IP addresses, like confusing game servers and the like, but with a bit of tweaking you could get it functional. I don't think this solution explicitly provides failover functionality, but I suppose that could be scripted in somehow.
pfsense [pfsense.org] is a nice turnkey solution for this too, if you're not into spending a couple weeks solid trying to make your debian or lfs distro act like a router.
db
Your load balancing "problems" have been solved (Score:4, Informative)
Can't load balance hosted services without a remote router? Round robin DNS with short TTLs, with a script to remove an IP if a link goes down.
Outgoing TCP connections are OK when using Linux:
http://lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS [lartc.org]
If you buy an off the shelf solution from the likes of F5 there's even more control.
Re:Point of failure (Score:4, Informative)
There's a bit more to it than that [...]
That is the understatement of the year :-)
The 'weight' feature is quite nice. It evens keeps the route selection sticky per-flow.
Re:Point of failure (Score:5, Informative)
Re:Point of failure (Score:3, Informative)
Systems are engineered for typical weather conditions. In California, heavy rain is sufficiently infrequent that utilities that utilities figure it's cheaper to fix lines after a storm. Similarly, new buildings in California almost always leak. If you're hit by earthquakes every month, it makes a lot of sense to invest in stronger infrastructure.
Re:Point of failure (Score:4, Informative)
Another alternative would be the Linksys/Cisco RV082 VPN router. THat too has dual-WAN support, and makes a nice home VPN endpoint as well.
Re:Dual WAN Router (Score:2, Informative)
But, you do raise another possibility: maybe the DSL company has some parallel telephone structure (OK, almost certainly not) on which they might offer bonding. That use scenario is explicitly mentioned in the linked Wikipedia article. This would be precisely the Article Poster's DSL/DSL idea, unlike many of the other, earlier, responses.
You're ignoring a key fact here: (Score:4, Informative)
The Japanese definition of "rural" is nowhere near the definition of rural here in the US. this is because they have an ungodly amount of people for the land they inhabit.
Basically, what I am saying is the Japanese idea of rural is, at best, like a marginally populated suburban neighborhood in the US.
Here are some raw numbers to better illustrate my point (from this study [columbia.edu], year 2000 numbers):
Japan total rural area (sq km): 273,646
Japan total rural population: 13,498,527
Japan rural population density (people/sq km): 49.32
US total rural area (sq km): 8,423,867
US total rural population: 54,936,968
US rural population density (people/sq km): 6.52
SEE THE DIFFERENCE? It's almost an order of magnitude! And the urban numebrs show a 3x difference between the US and Japan; closer, but still nowhere near each other.
Of course we have infrastructure problesm here in then US, and they largely don't; it just comes with the territory.