Multi site BGP Routing design

Justin Krejci jkrejci at
Mon Jun 8 18:42:41 UTC 2009

Thanks to all for the on and off list replies, they've been helpful.

We get full BGP routes from all upstream connections (currently they are all
different providers). The upstream bandwidth is cheaper at site 2 than at
site 1 and the private backnet connection is a fixed cost so when previously
considering the multi-ASN approach we would plan for each site using the
other as a transit/gateway using eBGP but put preference on sending out via
site 2 and maybe prepend site 1 AS on the local upstream SP so incoming
favors site 2 as well (we're already doing this preferential routing

I don't particularly care for the allow routes for our own ASN arrive from
an upstream BGP session especially when it seems like all carriers would
need to be cooperative on this, which may not be a big deal overall but adds
another layer of complexity and difficulty if we change/add/remove carriers
later on. What if they don't all support it, change their policies, or
upgrade to a new version of router code that makes the default/expected
behavior interfere.

I am thinking the multiple ASN route is the cleanest but the idea of letting
a default gateway (via static route maybe) out the local upstream connection
to reach the other site when the backnet link is down sounds like it would
work with minimal to no headaches but it just some how seems like a duct
tape job. Does this sort of technique have any significant flaws or concerns
associated with it?

-----Original Message-----
From: Adam Greene [mailto:maillist at] 
Sent: Saturday, June 06, 2009 8:38 AM
To: nanog at
Subject: Re: Multi site BGP Routing design

Hi all,

We actually have a very similar setup to what Justin asked about, with the 
exception that we advertise only some of our netblocks to one provider and 
the rest to the other. If one of the providers fails, we then advertise all 
netblocks through the provider which is still up. If the private link 
between our two locations fails, the two halves of our network communicate 
via the Internet.

>From what Justin described, I would think he would be able to keep a single

ASN and configure his network so that if the private link goes down, the two

newly disconnected halves of his network advertise only the netblocks they 
can still "see" (i.e. the ones on their half). As long as his internal 
network is set up with dynamic routing (iBGP / OSPF) the two halves should 
realize they have to get to the other half via the Internet.

In our case, we don't get full routing tables from our providers, just 
default routes. Perhaps in Justin's case something as simple as a floating 
static route via the Internet to the other half of the network would take 
care of any ASN weirdness. It doesn't sound like he really needs his border 
routers to speak BGP with each other while the private link is down. If he 
wanted to remove the BGP session entirely under these circumstances, he 
could do the iBGP peering between RFC 1918 addresses and thus force the iBGP

session to go down if the private link fails.


----- Original Message ----- 
From: "Saqib Ilyas" <msaqib at>
To: <nanog at>
Sent: Saturday, June 06, 2009 8:21 AM
Subject: Re: Multi site BGP Routing design

> For a given interconnection between the upstream ISPs for the two site, 
> once
> the direct link goes down, the time required for site A to learn the new
> route to site B and vice versa would be different with the different
> proposed solutions, right?
> Thanks and best regards
> On Sat, Jun 6, 2009 at 12:40 PM, Ivan Pepelnjak <ip at> wrote:
>> > To rephrase the OP's question, would it be BCP to acquire a
>> > second ASN, and without further de-aggregating, continue
>> > advertising each site's IP space to the DFZ, but from
>> > dissimilar ASs as opposed to the same one?
>> This would definitely be the best approach. You're not introducing new IP
>> prefixes and you're not extending AS paths, so the net effect on the 
>> global
>> BGP routing is zero (OK, you might have to use the 4 byte AS number :).
>> Just make sure that both ISPs you connect to allow you to advertise
>> "transit" prefixes. If site A public link goes down, but the private link
>> is
>> up, site B will advertise its own address space plus site A's address 
>> space
>> with an extra AS number in the AS path (and the upstream ISP might filter
>> that).
>> Ivan
> -- 
> Muhammad Saqib Ilyas
> PhD Student, Computer Science and Engineering
> Lahore University of Management Sciences

More information about the NANOG mailing list