Anycast 101

Mon Dec 20 12:28:43 UTC 2004

On 18-dec-04, at 22:31, Paul Vixie wrote:

>> i would be interested in hearing from anybody else who thinks that
>> turning on pplb in a eyeball-centric isp that has multiple upstream
>> paths is a reasonable thing to do, even if there were no anycast
>> services deployed anywhere in the world.

> so far, no takers.  i've heard from rfc-writers who say pplb was never
> meant to be used the way Iljitsch is describing it, and i've heard from
> equipment vendors who say their customers don't do that and that if 
> some
> customer did that and asked for support the response would be "don't do
> that!", and i've heard from network operators who say they would never 
> do
> that, and i've heard from customers of network operators who did that 
> with
> notable bad effects.

> but so far nobody has said "yes, what Iljitsch is describing should 
> work."

Apparently you also didn't get any pointers to RFCs or other 
authoritative sources that say "each and every packet injected into the 
internet must be delivered in sequence".

You feel you get to decide what other people should and shouldn't do. I 
find that dangerous. As long as there is no standard or law that says 
something can't be done, people are free to do it.

Apart from that, I'm not convinced per packet load balancing is as bad 
as people keep saying. In the absense of any research that I know of, 
my position is that per packet load balancing does have potential 
adverse effects, so per destination load balancing is preferred, but if 
there is a reason why pdlb doesn't fit the bill, pplb is a reasonable 
choice.

> let me summarize.  Iljitsch says that pplb is incompatible with 
> anycast,

No. What I'm saying in general is that anycast isn't 100% problem free, 
so:

1. There should always be non-anycast alternatives
2. It would be good if we had a way (= BGP community) to make sure that 
anycasted routes aren't load balanced across

I don't think either of these is unreasonable.

> since a pplb-using access router at the inner edge of an ISP could hear
> two different IGP routes to some destination, which ended up taking
> different exits from the ISP and thus different BGP paths.

I'm not even sure if I understand this sentence, but it sure doesn't 
look like something I said. What I said was, that if you inject packets 
towards an anycasted address into two different routers within a 
certain AS, there is a very real possibility these two packets will end 
up at different anycast instances. I'm on very firm ground here as this 
follows directly from the BGP path selection rules. (Although in real 
life this wouldn't happen too often because customers tend to connect 
to two routers in the same or neighboring pops.)

> whereas pplb
> would normally only operate on equal-cost paths, the BGP->IGP path 
> would
> hide the variance in BGP paths and make these "paths" eligible for 
> pplb.

Again: huh?

> i've said that pplb is only useful for turning two OC3's into an "OC6" 
> (or
> similar circuit bundling where a pair of routers has multiple 
> connections
> to eachother) and that even in this case, packet reordering is likely 
> to
> occur, which will make tcp-flow performance suffer across this "link".

But would the TCP performance over this "OC6 link" be better than that 
over a single OC3 link? That's the real question.

> i have also said that turning pplb on across non-parallel links, such 
> as to
> multiple providers or through multiple tunnels or whatever, would 
> pretty
> much guaranty that a word rhyming with "massive suckage" would occur.  
> and
> i've made these claims independent of anycast -- that is, life will be 
> bad
> if you use pplb outside its intended purpose, even if nobody anywhere 
> was
> using anycast.

Your argument is that since it's a bad idea to do this, nobody will, so 
making it even worse is ok. My argument is that even though it's a bad 
idea, some people will do it we shouldn't unnecessarily make things 
worse and/or make a reasonable effort to repair the damage.

> loath though i am to treat a "preponderance of assertion" as equivilent
> to "proof", i see no alternative on this issue.  noone is defending 
> the use
> case Iljitsch is proposing.  noone is even saying "i tried that and it 
> was
> OK".  lots of people are saying various things like "don't do that!" 
> and
> "are you crazy?"

And we all know that when you tell people not to do something they 
don't, and there are no crazy people connected to the net.