Anycast 101

Fri Dec 17 12:05:59 UTC 2004

On 17-dec-04, at 3:06, Steve Gibbard wrote:

>> under some very
>> specific circumstances it can also happen with per packet load
>> balancing.

> You're misunderstanding how per-packet load balancing is generally 
> used.

I wasn't saying anything about how per packet load balancing is 
generaly used, the point is that it's possible that subsequent packets 
end up at different anycast instances when a number of specific 
prerequesites exists. In short: a customer must pplb across two routers 
at the same ISP, and each of those routers must have different 
preferred paths to different anycast instances. This isn't going to 
happen often, but it's not impossible, and it's not bad engineering on 
the customer's or ISP's part if it does, IMO.

> Using per-packet load balancing on non-identical paths (in your 
> example,
> out different peering or transit connections) doesn't work.

That's right, because BGP only installs two or more routes when the 
path attributes are identical or nearly identical.

However, the attributes may be different (different next hop, IGP 
metric, MED) inside the ISP network but the differences can then go 
away at the next hop.

> Even when connecting to a unicast host, the packets would arrive out 
> of order,
> leading to some really nasty performance problems.  If anybody is using
> per-packet load balancing in that sort of situation, anycast DNS is the
> least of their problems.

Yes, this is why people are so terrified of per packet load balancing. 
Most of this fear is unfounded, though: the only way to get consistent 
out of order packets (a few here and there doesn't matter) is when the 
links in the middle are the same or lower effective bandwidth than the 
links at the source edge. And even then it will mostly happen for 
packets of different sizes.

> You appear to be assuming that every anycast server in the world 
> announces
> routes for every anycasted address.

No. I'm not concerned about what happens at the anycasted ends, it's 
the way it looks from any given vantage point throughout the network 
that matters.

> Are there scenarios where an
> outage would lead to a loss of all of the anycast clouds?  Of course, 
> but
> those scenarios would apply to Unicast servers as well.

The assumption is that it's universally benificial to see DNS addresses 
"close". While it is good to be able to see several addresses "close", 
it's better for redundancy when there are also some that are seen "far 
away", since when big failures happen, it's less likely that everything 
"close" _and_ everything "far away" is impacted at the same time.

>> Obviously this won't happen to the degree of unreachability in 
>> practice
>> (well, unless there are only two addresses that are both anycast for a
>> certain TLD, then your milage may vary), but even if 5 or 8 or 12
>> addresses become unreachable the timeouts get bad enough for users to
>> notice.

> Right, but if you're losing 5 or 8 or 12 diverse routes at the same 
> time,
> your problem probably has very little to do with anycast.

That's not the point. If without anycast this is better than with 
anycast, then this should go on the "con" list for anycast.