ICMPv6 rate limits breaking PMTUD (and traceroute) [Re: Comcast enables 6to4 relays]

Wed Sep 1 21:18:55 UTC 2010

Jack Bates writes:
> 1) Your originating host may be breaking PMTU (so the packet you send
> is too large and doesn't make it, you never resend a smaller packet,
> but it works when tracerouting from the other side due to PMTU working
> in that direction and you are responding with the same size packet).

Your mentioning PMTU discovery issues in connection with 6to4 prompts me
to confess how our open 6to4 relay has probably contributed to the
perception of brokenness of 6to4 for quite a while *blush*.

The relay runs on a Cisco 7600 with PFC3 - btw. this is an excellent
platform to run an 6to4 relay on, because it can do the encap/decap in
hardware if configured correctly.

At some point of the relay becoming popular (load currently fluctuates
between 80 Mb/s and 200 Mb/s), I noticed that our router very often
failed to send ICMPv6 messages such as "packet too big".

First I suspected our control-plane rate-limit (CoPP) configuration, but
couldn't find anything there.

Finally I found that I had to configure a generous "ipv6 icmp
error-interval"[1], because the (invisible) default configuration will
only permit one such ICMPv6 message to be generated every 100
milliseconds, and that's WAY insufficient for a popular router.
We currently use

     ipv6 icmp error-interval 2 100

(max. steady state rate 500 ICMPv6s/second - one every 2 milliseonds -
with bursts up to 100) with no ill effects.

Note that the same rate-limit will also cause stars in IPv6 traceroutes
through popular routers if the default setting is used.

The issue is probably not restricted to Cisco, as the ICMPv6 standard
(RFC 4443) mandates that ICMPv6 error messages be rate limited.  It even
has good (if hand-wavy) guidance on how to arrive at defaults - the
values used on our Cisco 7600 (and possibly all other IOS devices?)
correspond to the RFC's suggestion for "a small/mid-size device" *hrmpf*
(yes Randy, I know I should get real routers :-).

Anybody knows which defaults are used by other devices/vendors?

In general, rate limits are very useful for protecting routers'
notoriously underpowered control planes, but (1) it's hard to come up
with reasonable defaults, and (2) I suspect that not most people don't
monitor them (because that's often hard), and thus won't notice when
normal traffic levels trip these limits.
-- 
Simon.
[1] See http://www.cisco.com/en/US/docs/ios/ipv6/command/reference/ipv6_06.html#wp2135326