Anycast 101

Fri Dec 17 20:55:31 UTC 2004

On 17-dec-04, at 19:43, Paul Vixie wrote:

> i don't think iljitsch is in a position to teach an "anycast 101" 
> class.

If anyone feels they can do better, please step up...

> here's my evidence:

> note-- harald asked us to move this thread off of ietf@, so i've done 
> that.
> iljitsch added ietf@ back to the headers in his reply to me.  i'm 
> taking it
> back off again.  iljitsch, please leave it off, respecting harald's 
> wishes.

Hey! I missed this one. I'm on dnsop but it's pretty low on my to-read 
list.

Unfortunately, your evidence contains its share of errors so I'm not 
sure if you should be teaching the class either.

>> ... It's possible for bad things to happen if:

>> 1. some DNS server is anycast (TLD servers are worse than roots 
>> because the
>> root zone is so small)
>> 2. fragmented UDP packets or TCP are used as a transport
>> 3. a network is built such that packets entering it through router X 
>> may
>> prefer a different external link towards a certain destination than 
>> packet
>> entering it through router Y
>> 4. a customer of this network is connected to two different routers
>> 5. the customer enables per packet load balancing

> #1 and #2 are normal, even though fragmented udp isn't very common 
> nowadays.
> #3 is extremely common.  #4 is normal for high-end customers.  and #5 
> will
> only affect customers whose ISP shares an IGP with the anycast -- in 
> other
> words, "other customers of the same ISP".

Nope. Consider:

            +-------+   +-------+
            |ISPrtr1+---+ACinstA|
+------+---+---+---+   +-------+
|source|       |
+------+---+---+---+   +-------+
            |ISPrtr2+---+ACinstB|
            +-------+   +-------+

Where the anycast instances exchange routing information using BGP.

If there is no special BGP configuration in effect, the ISPrtr1 will 
prefer the path to anycast instance A and 2 to B, because the external 
path takes precedence over a same length path that's learned over iBGP.

The current Cisco multipath BGP rules require the whole AS path to be 
the same (which would be the case in this diagram if both anycast 
instances use the same AS number), but older IOSes only require the 
next hop AS and the path length to be the same.

>> Now the question is: how do we deal with this? I don't think removing
>> anycast wholesale makes sense and/or is feasible. Same thing for 
>> declaring
>> per packet load balancing an evil practice.

> as i said the other day, "all power tools can kill."  if you turn on 
> PPLB
> and it hurts, then turn it off until you can read the manual or take a 
> class
> or talk to an expert.  PPLB is a link bundling technology.  if you 
> turn it
> on in non-parallel-path situation, it will hurt you, so, "don't do 
> that."

Yes, per packet load balancing will cause reordering, and if that's an 
issue you shouldn't use it. But if with pplb packets end up at two 
different hosts, that's not the fault of the people who invented per 
packet load balancing or the people who turned it on, but the fault of 
the people giving the same address to two different hosts.

>> A better solution would be to give network operators something that
>> enables them to make sure load balancing doesn't happen for anycasted
>> destinations. A good way to do this would be having an "anycast" or
>> "don't load balance" community in BGP, or publication of a list of
>> ASes and/or prefixes that shouldn't be load balanced because the
>> destinations are anycast.

> since PPLB won't affect BGP (since BGP is not multipath by default), 
> this is
> not an issue.

If the uncommon network setup exists, and pplb is turned on, the 
problem can manifest itself. The fact that someone had to turn on a 
feature that's turned off by default is immaterial. (There is no BGP by 
default to begin with.)

>>> and they would know that PPLB is basically a link bundling 
>>> technology used
>>> when all members of the PPLB group start and end in the same 
>>> router-pair;

>> It doesn't make much sense to have multiple links terminate on the 
>> same
>> router on both ends as then both these routers become single points of
>> failure.

> i don't even know what conversation we're in any more.  why does it 
> matter
> whether they are single points of failure, if this is the 
> configuration for
> which PPLB was intended?

There is no requirement that all packets between two hosts follow the 
same path. So people who pplb have the IP architecture at their side, 
unlike those who implement anycast. So a little less blaming the victim 
would be in order. (Well, if there are any victims, because all of this 
happening is pretty unlikely.)