Transitioning Major Commercial Networks Between Providers?

Kalon asks: "I am the network manager for a large business park connecting to our countries largest telco/ISP. We've recently negotiated to move our Internet services to another major provider and I'm stuck with a problem: our range of IP addresses (8 Class C's) is half 'owned' and half 'leased' from of our current provider. With some 90 companies relying on our connection for continuous Internet uptime, how can I transition forward and reverse DNS services for those companies with websites/mailservers unlucky enough to be on leased addresses, and route mapping for those whole blocks I own? Considering I don't have APNIC membership, what tips do you have to speed or ease the transition, considering I have to deal with lumbering telcos who won't play nicely together, and 90 different companies all demanding Internet presence?"

Get into the fetal position and cry

[leviramsey]

...cause you're in for a hell of a difficult time.

The new provider ...

[RabidMonkey]

should provide some assistance.

I used to work in a provisioning dept, and we were tasked with some of this stuff, helping people, giving them contact information ...

If you're lucky enough, you'll get someone useful on the phone who should help you out.

Good luck ...

Buy 4 more Class C

[lewiscr]

Buy 4 more Class C subnets, and transition the people on the leased subnets now.

Then when you move, its just a matter of updating the routing.

Of course, changing the subnet routes could be another problem. A lot of the big routers don't propigate routes for anything smaller than a /19 subnet. I could be wrong about the size, but a /24 is right out. If your 4 existing subnets are all together, you've got a /22, and that might just work. Getting the 8 subnets together for a /21 would be better.

The truth about BGP routing of /24 networks

[Nonesuch]

A lot of the big routers don't propigate routes for anything smaller than a /19 subnet. I could be wrong about the size, but a /24 is right out.

Any ARIN registered, fully portable /24 will be reliably propagated through BGP.

Most any /19 netmask or larger blocks will also propagate, even if they are not "portable". The issue is the announcement in BGP of smaller subnets, from within ranges that were originally assigned by ARIN as a single large block.

IOW, an ISP with an assigned /16 might "sell" you a /24, and you might attempt to announce a route for this /24 via BGP through a different ISP. That announcement is likely to be filtered out by some backbone providers.

Set the TTLs LOW

[netringer]

As you get close tot the cutover time, set the TTLs (Time to live) on every DNS record you can get your hands on to a very low value.

Any host that does a lookup can legally only cache the answer as long as the TTL time. After that it has to ask again. With a low value you get a lot more requests to the DNS servers, but the host requesting will know about the change quickly so it'll find your servers at the new address as it changes.

The value is in seconds. 3600 would be an hour.

DNS and NAT

[funky womble]

If you have use of the old addresses for a little while after the new addresses are available, maybe you could run a NAT box on the old addresses for a short while so that inbound traffic could be directed to the new addresses. Maybe something like this -

1. setup the NAT box
2. change IP addresses on a machine
3. add NAT rule to direct traffic to new address
4. if more_machines_to_move goto 2
5. beer/pizza/etc, you earned it (:

Or instead of using a NAT box like that, you could possibly put it in front of customer servers and/or routers, so they could keep the configuration with their old addresses for a while longer, giving you a little more breathing space (obviously you need to update DNS while you're doing that - you might find tinydns, part of the djbdns package, useful as it can control the TTLs, making it fairly easy to expire DNS records at a particular time and replace them with new ones).

I know an ISP that has been putting off renumbering for several years, I don't think their former transit providers are too happy about having part of their own netblocks announced at them :/

Think of it as a challenge and also proof that there are certain (fairy limited) circumstances where NAT doesn't completely suck!

Re:DNS and NAT

[mrjohnson]

No way this will work. For normal traffic, yes. But it'll cause problems with several different vpn solutions. You don't want to be stuck helping almost a hundred different admins debug why app X suddenly stopped working.

I've tried it. :-)

Re:DNS and NAT

[funky womble]

The question specifically asked about websites and mailservers.

been there...

[[e-mail]]

Hello...

I've done this. It isn't fun, but it's doable.

quick options
1: use two providers
you get to keep your address
you have redundancy

2: hire someone that has done this before

long solution:
get your own address from arin or suitable authority. with 90 companies connected, getting a /20, /19, or /18 should not be a problem

bring up second connection. establish BGP sessions with both providors announcing all (old and new) ip addresses.

plan on transitioning a few customers per day. I imagine most of your customers are 9-5ers. They are easy. Save the more critical ones to last as you'll have had a lot of practice.

Using DHCP on their side helps

Co-ordinate with the companies IT person. You update the routers/routes, and they update the DHCP server. Easiest done as they leave for the day.

do not do NAT in the routers. It will kill your performance.

The whole transition could take up to two or three months. Don't get in a hurry. Do it right the first time.

Setup DNS servers on both old and new ip address blocks.

Consider keeping the old connection (maybe at a lower bandwidth) for redundancy. or kill old provider connection.

Easy, but tedious

[Gregg Alan]

Count yourself lucky that you aren't ALSO moving physical locations. Since you are only changeing providers you will *have* to bite the bullet and pay for duel service until you cut everything over. Since you have that much address space you must have some decent equipment and some good knowledge and experience.

Like some other posters have said, you are going to need to forward/NAT, etc whatever you move from one service to another until DNS catches up. But you will need to watch out for the idiot coder that used an IP instead of a domain. *THAT* will be the hardest part of this whole move for you.

You're joking right?

[anthony_dipierro]

You're the "network manager for a large business park" and you don't know how to do this? I suggest you go to your boss right now and tell him to fire you.

Re:You're joking right?

[nathanh]

You're the "network manager for a large business park" and you don't know how to do this? I suggest you go to your boss right now and tell him to fire you.

Why is this moderated funny? I can't imagine how screwed up the system is that allowed this person to be network manager when he doesn't even know how to transition a subnet. And he's asking for help on /. of all places!

Make sure they are portable.

[krangomatik]

Check and make sure that the IPs that you "own" are in portable address space. I've known companies that have contractually "owned" IPs, but they were non-portable, so it didn't really do them much good.

Some more details...

[trims]

First off, all the suggestions in the post above by chrismcc (here [slashdot.org]) are excellent. I am going to expand on them a bit.

First, you really should have an AS number and get a dedicated IP allocation from APNIC - it will be immensely useful, and save you a whole bunch of money and headaches in the future. There are some upfront fees (I just checked, and a /19 (32 class C) runs about $8192, with an AS number cost $500, both of which are one-time fees, and you don't have to be an APNIC member), but they're well worth it, and not excessive.

After you get the AS and IP block, talk to your new ISP and your old ISP. Advise them that they will need to be advertising your AS now.

If you don't already have it, upgrade all your border routers to support BGP, so you can actually use the AS number, and also so you can potentially use multiple different ISP pipes for redunancy and load balancing. Depending on your setup, this might cost some money. However, given your setup, I suspect that you don't have anything more than a Cisco 3600-series border router, which is fine (and the BGP upgrade isn't very expensive).

Using the new APNIC address block and AS number, number all your network equipment with the new IP space, keeping the original IPs from the old ISP in place.

Have the new ISP begin advertising the AS number and your new IP block. Do some testing from outside to make sure you can reach all segments of your network.

For all important machines and equipment (primarily servers, but stuff that generally retains a static IP), give them a new IP. Almost everything supports virtual interfaces now, so it's trivial to have two different IPs assigned to the same machine these days.

Do some more testing, to check that you can reach these machines via the new IP (do both internal and external testing, as required).

Update DNS to include both the new and old IP for all assigned machines. Change the TTL to something VERY LOW, like an hour or so.

Test DNS starting the next day, and do repeatedly for the next 3-4 days.

Update any servers providing DHCP or BootP or similar dynamic service to provide IPs from the new block. Continue testing.

After about a week of running both the new and old DNS entries in parallel with all machines assigned a NEW IP address, consider removing several of the DNS entries for the old IP space. Do this, and check to see if anything breaks. Fix it.

Remove all old IPs from DNS. Wait a week or so to see if anything lingering breaks, and see if you get any reports from outside as to problems.

Remove all of the old IP numbers from all machines and network equipment, and notify the old ISP that you have completed renumbering, and that they should change routing so that IP space no longer points to you. Inevitibly, you probably will run into something you missed. Fix it.

Ditch the old ISP, or keep them for redundancy/load balancing, but make sure they're publishing your AS number and new BGP information.

Many of these steps above will require coordination with your client companies. However, if you do it right, there should be NO DOWN TIME, and the transition will be transparent to your clients.

-Erik

What the hell? Are you serious?

[Wakko Warner]

Even at the shitty places I've worked, there have been at least two uplinks to major ISPs, with BGP running to enable redundancy and increase network efficiency.

I now have three questions for you:

1. How "large" is your "business park"?

2. How do you explain even insignificant outages to your tenants now?

3. Why don't your customers have their own uplinks?

That you're even in this situation scares me. Good luck finding a way out.

- A.P.

Now you ask us.

[KurdtX]

Hmmm, great timing, make the switch without knowing how you're actually going to handle the details, and then roll the dice and hope the slashdot editors will post your story.

I'd say this would be pretty much equivalent to: Hello slashdot, I'm a Windows admin who is responsible for a large fortune 500 company. I just placed an order for enough machines to completely switch our network over to linux 'cause I heard it's so 'leet. With some 9000 developers who will need to continue their work, how do I find software that they can use? Considering I'm not a member of any LUG, what tips do you have to ease the transition, considering I have to deal with lumbering admins who also have never used linux, and 9000 developers all demanding their ability to do work?

You do realize that not thinking this through may bring your company down, as well as many of those whose access you manage. Ok, maybe it's not as bad as I'm making it sound, but this "minor detail" probably should have been thought about, tested, and then re-thought about before you agreed to do the transition. It's great that you may be the fastest mover in your area, but if you don't spend enough time looking around, you're going to run into a brick wall soon enough.

Well, Here's what we did...

[DenniRuz]

We leased 5 class C's and wanted to migrate to another carrier. What we essentially did was get a big line to the new carrier and downgraded the line to our old one to a 128k fractional T. That let us keep the leased addresses. Then we set up BGP to advertise our the addresses we were using directly and that let us move IP's at our convenience..