Multihoming Suggestions w/o at Least a /24? 55
An anonymous reader asks: "I work for a small company who is looking to get a multihomed Internet connection for redundancy.
The logical conclusion would be to get another internet connection to another provider. However, in the case of a primary connection failure, we need to be running BGP to have our internally-hosted sites still accessible to the Internet via the 2nd connection. The problem is that we only have a /28 (16 IPs), which is too small to make it past most route filters, and would then mean that we still couldn't be reached if the primary T1 is down. So, what's our options? (and no, lying and getting a /24 isn't a valid choice)"
It depends on the services... (Score:4, Interesting)
Re:It depends on the services... (Score:5, Informative)
Yeah outbound traffic is easy, it's the inbound he's having a problem with I'm sure. The problem with two sets of addresses and DNS switching is the caching. Even if you set your records to expire in 30 seconds or something crazy like that, at various levels the records *will* get cached much longer than that, and it will "problematic" at best.
This question is truly worthy of Ask Slashdot, which is a first in a long time. I have yet to see a good answer for someone who wanst truly redundant internet connectivity and has too small an address space to really do BGP peering.
I thought of one solution at the ISP end of things, which would require partnerships between ISPs. Two distinct competing ISPs could grab a decent-sized netblock and share it. They sell these IPs to customers wanting dual-homed access from both ISPs, and split the money. In this type of scenario the customer can BGP to both ISPs, who in turn BGP with each other and the real backbone, and you can get all the redundancy you need in case of ISP or wan-link failure.
Re:It depends on the services... (Score:2)
I was thinking that you could also do it on the cheap by running 2 lines with your typical outgoing redun links. Which is fine.
Then have a secondary DNS located outside your network, with very very short TTL and a perl script that checks your link state. Once one group of IPs go down your secondary remote DNS outside your network starts to feed your redundant group of IPs.
I do something like this with my home cable ip which is DHCP'ed. On my box at the house I have a script that check for ip changes, if it does change it notifies the remote DNS to change to the new ip. I don't see why you could not do something like this with 2 cheap links and small blocks. You would just have to worry about convergence and proxys that feed to your sites.
Anway, just leaking at the brain...I have been known to be wrong.
Re:It depends on the services... (Score:1)
Years ago you could go to the internic (Score:1)
but this was back in the 1995-6; things may be different now.
Vaguely Helpful Comment (Score:2)
Legal v.s. technical issues. (Score:4, Informative)
Of course, the usual question is: what can you afford to have redundancy ?
Because before technical solutions, you might want to review the contract with your access provider to include liabilities. The contract itself might cost more, but it might be simpler than a real redundant solution.
Because unless you know for a fact than your access provider is not reliable and has bad support, playing the redundancy game might be a bit more expansive than "simply" getting a double connection from the internet.
Let's do the excercise: you want a dual internet connection, that's OK, but you surely do not want a single router=single point of failure. So you have to buy another router, most probably the same brand as the one you already have, so to be able to use the (most probably) proprietary high availability solution. Provided your current model supports HA, or you will have to buy a more expensive one ?
Which brings to mind that having a redundant link (with an SLA :-) from the same provider might be an excellent idea, since they are probably aggregating your /28 to other /subnet, your route advertisment won't get lost in their network until it gets aggregated. Just make sure it does not get aggregated on the next hop ;-)
Well, if you are willing to pay for multi-homing, woul'dt it be easier to try to obtain an SLA with only one access provider, SLA including an redundant routing connection, with some redundancy protocol handled
Re:Legal v.s. technical issues. (Score:1)
Re:Legal v.s. technical issues. (Score:2)
Multihoming to dual providers provides a level of redundancy not possible with a single provider - you get multiple backbone access points.
And yes, it's expensive, but sometimes that cost is worth it (not everything is monetarily measurable - not every organization is a business).
Re:Legal v.s. technical issues. (Score:1)
You really want to do multi-homing ? Yes, you sure ?
Well, then get two different provider with a different set of addresses, set up two routers for each one of your feedm and set up your servers to have two differents IP addresses per machine. Of course, to be fully redundant, you have to register two different DNS domains (on, in .com and one in .org will definitely look k00wl).
What ? It does not work ? assymetric routing ? That's for whinnies.
Be real dedundantm duplicate everything, (but your boss: this one should be replaceable).
Fake it with DNS? (Score:2, Interesting)
If the main link goes down, so does the primary nameserver. The secondary nameserver (on the backup link) then returns IP's that are routed through the backup link.
This should work, but it probably goes against several RFC's..
Re:Fake it with DNS? (Score:1)
Of course, if they have similar amounts of bandwidth, etc., you might actually consider this to be a good thing :-).
It's still not going to work great for fail-over, though (thanks to things like DNS caches). As distasteful as it might be, the most cost-effective way for you to approach this is probably going to be with an SLA (service-level agreement) with your ISP.
Re:Fake it with DNS? (Score:2)
hehe.. That's my favourite quote. The tasks I'm given at work require that I say that every other day. I thought I was the only one.
Re:Fake it with DNS? (Score:1)
"RFC's - Rules For Competitors"
Re:Fake it with DNS? (Score:3, Informative)
should round-robin the IPs from each link on both
servers. Only drop the IPs from link1 if link1
goes down. Then even if there is some dead cache
on the network, at least your clients can reach
the server by trying again.
Re:Fake it with DNS? (Score:2)
Re:load balancer (Score:1)
The Radware [radware.com] LinkProof and FireProof work very well.
Faking it is your only option. (Score:3, Informative)
First, find an upstream ISP that is multi-homed to your satisfaction. Buy some IP's from them and put in a router or two for redundancy.
Next, build two or more tunnels to the ISP over different circuts or providers and run your own small BGP network on private IP's between the router at your multihomed isp and the routers on either end of your connection. Assign the IP's that belong on the multihomed network locally and let your own routers run BGP (or OSPF or whatever else you want to use instead like load balancing) between your LAN and the multihomed network.
It's hackish. It will be fairly expensive. It will also, however, work, let you keep your servers on-site, and give you greater control over redundancy and failover than you'd get with two upstream providers allowing you to use BGP anyway.
In the end, it might work out to be cheaper to do this in the long run since you wont have to pay any upstream ISP for letting you do BGP. You'll just have to pay for colocation somewhere, which could be a lot cheaper.
~GoRK
Re:Faking it is your only option. (Score:1)
OSPF will do the job just fine, and provide reasonably quick convergence too.
Re:Faking it is your only option. (Score:2)
~GoRK
VRRP is a possibillity. (Score:3, Informative)
The option is Virtual Router Redundancy Protocol (VRRP). A brief description of VRRP, including a diagram, can be found here [cisco.com]. Keep in mind that there are numerous other manufacturers that support the VRRP standard, you don't *have* to go with Cisco. Also, remember that with VRRP there is still a single point of failure, the ISP. This means that your ISP had better be a good one.
Re:VRRP is a possibillity. (Score:1)
1) At least 2 independant fiber connection that use geographically different routes to get to the building.
2) Dual redundant UPS power.
3) At least 1 generator.
4) A/C should run on UPS or Generator power.
5) A fully redundant network(mesh topology?).
After you find an ISP with ALL of these you need to get a connection on each of the incomming fiber connection. As far as your end of things, 2 routers would be good, but you can pick up something like an OLD and inexpensive Cisco 75xx or Bay BCN with dual serial or T-1 cards and DUAL CPU's as well as at least dual power supplies.
Reliable connections don't have to cost an arm and a leg, just a LOT of research and careful planning!
Good luck!
Reverse proxy off site? (Score:3, Informative)
Its something i have intended to look into for work, as it would jsut be a extension of what we currently use for firewalling anyways, port 80 is redirected from the gateway to a machine behind the firewall. To carryout a port 80 redirect on two publically available ips is probably jsut as trivial, infact as ive been thinking this through, i have tried it with OpenBSD on both ends and apache as the webserver, a rdr on the outside box gets my webpage fine.
Mutlihome to a Single ISP (Score:1)
Try to get a larger address block anyway (Score:1)
Dual IPs for everything (Score:2)
I know it's not strictly necessary to do the redundant hardware/network thing, but why not? It's only a few $20 network cards and a switch.
If you have multiple MX records (one per IP), then you won't lose any email either.
It would be nice to dual-home, but dual IP is a workable solution for the small business.
--Mike--
Re:Dual IPs for everything (Score:2)
Re:ARIN (Score:1)
Rain Connect (Score:2)
We evaluated their software, but since we have a
Re:Rain Connect (Score:2)
Rain Connect [rainfinity.com]
You are entitled to a /24 if going multihomed. (Score:2, Informative)
Hello,
There is no need to lie. Going multihomed is reason enough to request and obtain a /24 from one of your two providers, despite the fact that your network size only requires a /28. I have performed this exercise for companies of your size many times over, and trust me, any major network provider will give you a /24 if you are switching over to BGP and getting a second connection.
The effect of imposing a /24 or greater limit on BGP routes is that providers need to be more sensative to the needs of companies who, when considering network size alone, can't justify a /24. Thus, going multi-homed is enough of a qualifier by it self to obtain a /24 from an upstream ISP.
-James
Suggestions for j00 (Score:2)
Have you asked them?
Based on the information provided, here's my suggestions:
1) Are sattelite connections doable for you? It can be expensive but if you're company needs a quick fix until the main T1 comes back up, this may work for you.
2) Would it be possible to swtich your company's failed T1 through a secondary route to the internet? Most T1 Providers offer this.
3)DSL. I know this may seem wierd but in the event of an outage of service, you need to provide the bare minimum of connectivity to your customers internaly and externally.
Dolemite
Why not a /24 block? (Score:1)
You do have (a) firewall(s), don't you?
At a minimum, that's already 25% of the existing address space in use, without including any servers you may, or NAT addressing you may be doing.
Break your
This scheme gives you just 5 useable addresses left, assuming you've got two routers and a firewall. The available
With 11/16 addresses used, you're at 68% of your address space. One more host puts you over the top.
Use this type of addressing to get the better of the two ISPs you choose to delegate a routeable block to you.
Obtain from ARIN, an ASN. Inform the other ISP that you want to BGP peer with them, tell them if you want full routes, or summary routes only advertised to you, and give them your ASN and the IP addresses of the router(s) they'll be peered with. Give the first ISP the same info. Both ISP's should give you ASN's and peering info for themselves as well. Here's the catch... make sure you don't turn yourself into a transit area between the two ISPs. Filter your BGP adverts so that you're only advertising routes originated in your AS.
Simple, huh?
Joe
Security (Score:2)
And just about anybody can get a wireless link these days.
Damn, I'm feeding a troll, aren't I?
Radware Linkproof (Score:1)
Magic Radware box... check out the Linkproof or Linkproof LT depending on your needs. Dynamically add or drop providers seamlessly, use your exisiting IP's, handles the failover through a dynamic DNS system onboard.
Here's how I'm actually _DOING_ it cheaply (Score:1)
After much searching and testing I built my router using FreeBSD and IPFW. More on that further down.
Each ISP has given me a block of addresses from their CIDR block. I multihome proxy servers and email servers for inbound and outbound connections. They have one interface with multiple IPs bound. Nothing special there. Their default route is my FreeBSD router.
The freebsd router has mulitple ethernet interfaces. One per ISP and one for my servers. The ISP-facing interfaces have
The default route on the freebsd box is one of the providers.
I use IPFW for egress routing. Packets on the OUT side of the interfaces facing the ISPs are checked for source addressing and either allowed through or pushed over to the proper interface. Works like a charm.
interfaces:
em0 aa.bb.dd.128/25 (my side)
em0 xx.yy.zz.192/27 (also my side)
em1 aa.bb.cc.220/30 ISP A
em2 xx.yy.zz.188/30 ISP B
the rules I use:
ipfw -q add 100 fwd xx.yy.zz.189 ip from xx.yy.zz.192/27 to any out xmit em1
ipfw -q add 201 fwd aa.bb.cc.221 ip from aa.bb.dd.128/25 to any out xmit em2
There's also ipfilter in there handling filtering. IPFW only handles the egress routing.
DNS fills the gaps. I return at least two A records for the hosts I publish.
I used Linux for a short time in this router function but got bored with problematic network drivers.
That Radware device and the one by F5 are doing the same thing, but for at least 5 figures. I looked at them and then opted for this cheaper solution. I just bought duplicate router hardware and just keep a cold spare.
ARIN has propoals before them, too. (Score:1)
here's a hackish proposal i've made... (Score:1)
This applies to web servers as that is my primary concern, but could be used for other types of services.
Setup 4-10 front end web servers and do round robin DNS between them for your site. The 4-10 all need to be colocated on differnet parts of the Internet, differnet providers all in data centers is ideal. These sites can be load balanced/failover protected for more $$$ of course. These are your initial contact points for a client, and they have either your home page or a forward to your currently active site(s). This speeds up your ability to rapidly change from your live datacenter for your real site to your backup datacenter. You run a monitoring / mangament program that's constantly communicating to these servers which is the current live site. (this part of the solution must be VERY secure otherwise you're going to reduce downtime) Very secure and also automated fast, with a safe default to fall back on, like a local static page. Now if your main site gets hit by a 500MB/sec tcp/udp flood on random ports from randomly spoofed ip addresses (like a certian site I manage did 2 months ago), you simply move the current live site to backup site 2, or 3, or 4. If the attack is directed at your dns then it has 4-10 times the total networks to flood and should receive substantialy more attention than were it to take down only 1. If you have 10 front end sites up and one gets taken down then 10% of the people asking for DNS are going to get the wrong site, at least until you update your DNS.
Anyways this is a proposal to a problem made in hind site, but it's one we're working on implementing.
best of luck to you! If there is some poor mans BGP out there I've love to hear about it.
in re-reading this I realize it doesn't seem directly applicable to the needs (cost, etc) but if you were to use 1-2U per colocation facility and less than 1MB/month each you could pay in the few hundreds per front end site, and keep your local facility....
I run Linux on my HP Laptop [submarinefund.com]
The VoyNetworks Solution to redundancy (Score:3, Informative)
We want 0 downtime. There's no way to guarantee that any equipment is without failure. Something can/will always break. That's something you have to accept.
voyeurweb.com is located in colo facilities in both New York and Tampa. Each facility has it's own network drop. The size doesn't matter, but for reference, it's 1000base fiber in each location.
We have at least 5 machines in each location. Each machine has it's own IP, and in some cases multiple IP's just to increase it's load (faster machines can handle heavier loads).
You put multiple A records in your DNS. When a customer browses to your site, they get any one of the IP's randomly. Here's what an 'nslookup' returns for voyeurweb.com
> nslookup voyeurweb.com | grep Address
Address: 63.208.2.23
Address: 63.208.2.25
Address: 63.208.2.62
Address: 63.208.2.64
Address: 63.208.2.84
Address: 63.208.2.97
Address: 209.247.59.14
Address: 209.247.59.15
Address: 209.247.59.16
Address: 209.247.59.17
Address: 209.247.59.84
Address: 209.247.59.85
Address: 209.247.59.86
Address: 209.247.59.87
The 209.247.59 IP's are in New York. The 63.208.2. IP's are in Tampa. We're favoring the New York network a little bit, because we have some other specialized sites running in Tampa, and want an equal load between the cities. Right now, we're pulling about 450Mb/s per city at peak time. Just half of our 1000Mb/s drop. We've just added 1Gb/s fiber in Los Angeles, to increase our redundancy. How redundant you make youself is really up to how much the bosses want to spend, and how safe you want your site. Like I said, we want 0 downtime, and we achieve it.
The nice part is, if a machine fails, the client hangs for a few seconds, and then goes off to the next IP. If all the IP's in a city fail, the client can potentially hang for up to 30 seconds, before going to a server that works. Your browser will continue to use any IP that works.
We use a relatively short ttl in our DNS records, so if we decide to shut down all the servers in a city for any reason, within an hour all the traffic stops to them. I've done this many many times now, I'm 100% sure it works.
If we have a known problem (say a server has a hardware failure), you take it out of your DNS, and within an hour, no one is even trying to hit it.
We've done this with pairs of machines, or in the case of voyeurweb.com, up to 25 machines.
It's so simple it shouldn't work. I've been told by quite a few people that it won't work, but then I'll prove to them that it does..
Before we did this (before I was admin), if a machine failed, thousands of viewers would write in complaining immediately. Now, I can take a few machines down for maintaince, and no one notices. If we have a mystery crash at 4am, it's not fatal, we fix it when we can..
If you're a viewer, you probably didn't notice that we shut down all of Tampa for voyeurweb.com, for a week because of provider problems (lack of available bandwidth). You probably didn't even notice when we swapped out all the New York servers.. We were polite with some of them, but got bored with it, and just started yanking cables after a while.. We didn't receive a single Email asking why the sites were down. When a server went down, we did notice an increased load across the rest of them. It's nice having a *LOT* of servers running. If you have 10, you only get a 10% load increase across all the other servers when one goes down.
We don't depend on BGP. We don't depend on expensive load balancing equipment. We don't depend on anything other than the fact that people use browsers, and they resolve IP's through DNS.
In your case, you should have two ISP's providing you bandwidth. Each ISP should issue you a block of IP's from their available pool (like, it's hard to be on the Internet without it).. I'd say, if you want a site that stays up, set up a pair of mirrored servers. Give the first one an IP and gateway of the first ISP, and the second one an IP and gateway of the second ISP.. I could name off over a dozen sites that do this now, but I won't.
If you want to get real fancy, get one machine, put two IP's on it (one from each provider), and have a script monitor each gateway. If one fails, switch to the other. But this doesn't do anything for redundancy if one server should fail.
I hope I've explained this well. I've never seen it well documented anywhere. It's in the BIND documentation somewhere, but they have a real convoluted method of CNAME's and A records, which other documentation says are completely against some of the RFC's, so you shouldn't do it.
Even our evil nemesis does it...
>nslookup microsoft.com | grep Address | sort
Address: 207.46.134.155
Address: 207.46.134.190
Address: 207.46.134.222
Address: 207.46.249.190
Address: 207.46.249.222
Address: 207.46.249.27
Two different networks, splitting the load.
If CNN does it, it must be good.
> nslookup cnn.com | grep Address | sort
Address: 64.236.16.116
Address: 64.236.16.20
Address: 64.236.16.52
Address: 64.236.16.84
Address: 64.236.24.12
Address: 64.236.24.20
Address: 64.236.24.28
Address: 64.236.24.4
BTW, yes we get constant DoS attacks against us.. Sometimes I entertain myself by watching the logs.
To the script kiddie that "took down" one of our machines the other night.. Ummm, you didn't. I was annoyed at seeing the logs, and dropped all traffic from your network.
Re:The VoyNetworks Solution to redundancy (Score:1)
At the time I only tested with IE but Mozilla may do the same thing. I don't know.
Re:The VoyNetworks Solution to redundancy (Score:2)
It's possible, but when we change our DNS, users stop hitting it after an hour.. It could be that the users open and close their browser frequently, or the newer browsers are better behaved.
Re:The VoyNetworks Solution to redundancy (Score:2)
Probably would work for intelligent browsers.
Works for most apps and users - since most people have been trained to "reload/retry/reboot"...
It works for some telnet clients - they try the alternate IPs.
That's not a redundant internet connection (Score:2)
It's simply round-robin DNS to a bunch of web servers. It's a reasonable technique for your goals, but it's got nothing to do with the original question, which was how to keep his company online when one ISP/connection/link fails, and to both outgoing and incoming traffic of all kinds find the right place to go.
Re:That's not a redundant internet connection (Score:2)
Actually, it should work fine for him..
With two servers, and IP's on both networks, even if they're both on his same local network, he'll remain up, regardless if one of the lines dies..
The only concern that I could immediately see, would be more complicated stuff, like database work. But, that's easy enough, since he can tell each machine that both networks are local, so he doesn't go outside of his network for the connection.
cable modem and ADSL (Score:2)
We have two broadband links to our small downtown office. Each of these links terminates at an OpenBSD firewall. We briefly looked at what kinds of fail-over models being multihomed in the DHCP ghetto makes available to us (actually our ADSL service provides two static IP addresses, but they still grant you those addresses via DHCP leases).
We can do some cool things with Zebra (OSPF) over our VPN (all our offices / developer's have their networks in different portions of the 10.x space). If segments of our VPN fail, the traffic would find other segments forming a viable route to the final destination. Within the VPN we can pretend that we have a real network.
Fail-over to the outside world is more difficult.
One issue which no-one has mentioned here is stateful firewalling. Even if we can broadcast route information to the two OpenBSD gateway machines, it will break stateful firewalling completely. That said, most of our traffic is not session oriented. A broken HTTP request will fail-over at the application layer. No biggy.
The primary form of session oriented traffic we use is OpenSSH tunneling. The simple solution here is to remove stateful firewalling on the SSH port. I have to doubt that stateful firewalling your SSH ports accomplishes much.
For fail-over of our public web servers we decided against nasty DNS schemes. You don't get load balancing (just load sharing, which is not the same thing), there are issues about DNS record caching, your control is poor, when problems arise your ability to investigate the problems is all in a muddle. We don't tolerate that kind of thing here.
Instead we are directing all public URLs to a highly reliable and well connected public hosting company. That host is returning a frameset document containing a single frame window, with the embedded frame directed at one or the other of our two broadband services. Our firewalls are sending pings to the public hosting company so that the PHP script there can decide which of the two URLs to send out on each request.
There are plenty of extra complications since we are also thunking requests into different SSL compartments and we are using the OpenBSD firewalls to run chroot Apache in proxy mode (which allows them to filter the URLs before they reach the internal content servers).
With the right magic additions to the frameset code we can make the public URL track navigation through our content without revealing the ugliness of our internal fault-tolerant, SSL compartmentalized URLs.
Framesets suck, but not nearly as much as borking the DNS standard.
We are fortunate that our client base is narrow and we can impose client browser requirements. Our traffic volumes are low enough that we can afford the overhead of trampoline packets on each URL access.
We achieved a fairly low standard of fail-over in the end, but the good thing is that we didn't have to involve anyone else, and the fail-over we do have is sharply defined and easy to diagnose.
Forget it, its not happening (Score:3, Insightful)
That idea went out the window with the introduction of CIDR.
There were good reasons for this, primarily the unmanageable growth of route-tables.
IPV6 will never see the light of day because if IPV4 can't be routed economically out to the edge of the network, then increasing the address space by a large factor will not help matters.
There is no way around this except by removing CIDR for a decent proportion of the internet, but route-tables will of course baloon hugely.
So, while 'the people' want to be able to multihome etc. 'the backbones' don't want to have to scale up their routing capacity by a large factor.
Which all boils down to the conclusion that the powers-that-be on the internet have decreed 'thou shalt not multihome unless thy pockets are extremely deep and thou has at least a
This is unfortunately the way it is, and won't change under the current 'internet regime'.
Don't Forget... (Score:2)
What would BGP buy you? Well, if you ran your second T1 to another CO and to another ISP POP it would let you survive an entire CO or POP outage. But, how often do those happen with a good ISP? Almost never. How much would it cost to run that T1 to another CO? A ton.
I've already been down this road and found out that a dual T1 setup with something like CEF takes care of the job, unless you want to spend a fortune.
Also... (Score:2)