ICMPv6 rate limits breaking PMTUD (and traceroute) [Re: Comcast enables 6to4 relays]

Wed Sep 1 21:50:28 UTC 2010

On Wed, 01 Sep 2010 23:18:55 +0200
Simon Leinen <simon.leinen at switch.ch> wrote:

> Jack Bates writes:
> > 1) Your originating host may be breaking PMTU (so the packet you send
> > is too large and doesn't make it, you never resend a smaller packet,
> > but it works when tracerouting from the other side due to PMTU working
> > in that direction and you are responding with the same size packet).
> 
> Your mentioning PMTU discovery issues in connection with 6to4 prompts me
> to confess how our open 6to4 relay has probably contributed to the
> perception of brokenness of 6to4 for quite a while *blush*.
> 
> The relay runs on a Cisco 7600 with PFC3 - btw. this is an excellent
> platform to run an 6to4 relay on, because it can do the encap/decap in
> hardware if configured correctly.
> 
> At some point of the relay becoming popular (load currently fluctuates
> between 80 Mb/s and 200 Mb/s), I noticed that our router very often
> failed to send ICMPv6 messages such as "packet too big".
> 

That potentially starts to explain why I haven't noticed PMTUD issues
on my 6to4 tunnel that I've been running for a number of years, and have
been quite surprised be and somewhat doubtful of people to say there are
issues. I've pumped the MTU up to 1472 on it to suit my PPPoE 1492 MTU,
instead of leaving it at 1280, and haven't had any issues that have
made me suspect PMTUD. I'm in Asia Pacific so I've probably either been
using 6to4 gateways that are lightly used, or ones that have had this
parameter changed.

> First I suspected our control-plane rate-limit (CoPP) configuration, but
> couldn't find anything there.
> 
> Finally I found that I had to configure a generous "ipv6 icmp
> error-interval"[1], because the (invisible) default configuration will
> only permit one such ICMPv6 message to be generated every 100
> milliseconds, and that's WAY insufficient for a popular router.
> We currently use
> 
>      ipv6 icmp error-interval 2 100
> 
> (max. steady state rate 500 ICMPv6s/second - one every 2 milliseonds -
> with bursts up to 100) with no ill effects.
> 
> Note that the same rate-limit will also cause stars in IPv6 traceroutes
> through popular routers if the default setting is used.
> 
> The issue is probably not restricted to Cisco, as the ICMPv6 standard
> (RFC 4443) mandates that ICMPv6 error messages be rate limited.  It even
> has good (if hand-wavy) guidance on how to arrive at defaults - the
> values used on our Cisco 7600 (and possibly all other IOS devices?)
> correspond to the RFC's suggestion for "a small/mid-size device" *hrmpf*
> (yes Randy, I know I should get real routers :-).
> 
> Anybody knows which defaults are used by other devices/vendors?
> 
> In general, rate limits are very useful for protecting routers'
> notoriously underpowered control planes, but (1) it's hard to come up
> with reasonable defaults, and (2) I suspect that not most people don't
> monitor them (because that's often hard), and thus won't notice when
> normal traffic levels trip these limits.
> -- 
> Simon.
> [1] See http://www.cisco.com/en/US/docs/ios/ipv6/command/reference/ipv6_06.html#wp2135326
> 