oh k can you see

Tue Nov 1 18:48:33 UTC 2005

On Tue, 1 Nov 2005, Daniel Karrenberg wrote:

> We are considering to add a covering prefix announced from global nodes
> relatively quickly.  This should solve the particular problem and we
> cannot (yet) see any problems it would create. But this is more complex
> than the current state and thus brings us further away from salvation ;-).
> If there are reasons not to do this, please let us know.
>
> We are also considering seriously to treat "local" nodes and global
> nodes the same routing wise wherever possible.  This will be done
> one-by-one with the proper announcements and concurrent measurements.
> My personal hope is that we can do this for all K nodes and thus remove
> all BGP cleverness that originates from us. This does not mean that all
> cleverness concerning K's routes would be removed though.

Here's what we do on the PCH anycast network, to derive our "anycast 
cleverness" from the network topology rather than from BGP hacks.  It 
seems to work.  Other ways of doing this are presumably valid as well:

We've got four global nodes (nodes that have transit, rather than just 
peering), in Hong Kong, Palo Alto, Ashburn, and London.  We get transit 
from the same transit providers in all four locations.  Our transit 
providers hot potato to us, so as long as their peers hot potato to them, 
those who can't get to one of our local nodes should get to the 
topologically closest global node (topology, of course, does not always 
match geography).

We've then got a much larger number of local nodes, which look just like 
the global nodes except that they don't have any BGP transit.  They're all 
at exchange points, they all peer openly, and we encourage our peers to 
peer with us at all common locations and to treat us like a normal peer. 
That means they don't announce us to their transit providers, but do 
generally announce us to their customers.  Since this is the same thing 
that pretty much every network that's peering either globally or locally 
does, this doesn't require anything non-standard or hackish.

Our peers and their customers see us at whatever set of nodes they connect 
to.  Those who don't peer with us, and aren't customers of any networks 
who do, see us at a more limited set of locations.  This does mean we have 
to turn down offers of donated BGP transit from time to time, and we did 
have to turn off one peer who decided we were a good cause and was 
determined to give us transit whether we wanted it or not.  There have 
been a few times when we've found our routes being leaked (fortunately by 
networks with considerably longer AS paths to most of the world than our 
transit providers) and have had to turn down sessions until the filters 
got fixed.  These are the same issues we had at a real intra-connected 
global network I used to work for; it's not anything special about 
anycast.

The cases of suboptimal routing we see this leading to generally stem from 
networks that are unwilling to peer with us in their home markets but are 
eager to peer with us somewhere else.  Their generally suggested way 
around this is that we should buy transit from them, and our response is 
that we aren't going to pay them to accept free services from us.  Again, 
that's really a standard peering politics issue, and has nothing to do 
with anycast (and we're still generally closer to them than we would be 
with an arbitrary unicast location).

The remaining theoretical concern that might be solved by NO_EXPORT would 
be a situation where a network closer to one of our global nodes was 
getting transit from a poorly connected network close to one of our local 
nodes.  However, simple economics works against that.  Connectivity to or 
from poorly connected regions tends to be really expensive, so it's 
unlikely that anybody is going to be hauling traffic over those links when 
they don't have to.

My impression (and I think this is what Bill was saying yesterday as well) 
is that most of the weird routing problems that come up with anycast are a 
result of treating anycast as something different and special, which it 
doesn't need to be.

-Steve