Packet Loss on Level3

Fri Aug 21 18:07:19 UTC 2009

On Fri, Aug 21, 2009 at 10:07:44AM -0400, Dustin Schuemann wrote:
> Is anyone else seeing packet loss on Level3.
> 
>  6. ge-6-11-137.car2.Detroit1.Level3.net                  2.9%    35   
> 372.1 148.6  19.4 704.8 127.4
>  7. ae-11-11.car1.Detroit1.Level3.net                     8.6%    35   
> 268.1 161.5  21.3 691.8 156.0
>  8. ae-8-8.ebr2.Chicago1.Level3.net                       0.0%    35   
> 173.6 142.8  34.5 532.4 117.1
>  9. ae-21-52.car1.Chicago1.Level3.net                     5.7%     
> 35   78.1 210.3  35.7 631.2 157.9

If the loss doesn't continue for every hop along the path, it isn't real
loss. For example, if there was actually a problem at hop 6, it would
also affect the packets that are traveling to hop 8, so hop 8 wouldn't
come up loss-free. Of course it's always possible that there is loss on
a particular circuit that is part of a link-aggregate bundle, which
causes only certain packets hashed certain ways to go down the "bad"
channel, but typically you wouldn't see any loss at all since it's so
hard to target that one bad channel channel.

The far more likely scenario is that you're seeing routers with
overloaded control-plane or exception processor rate-limits, due to
excessive icmp generation demands. Running mtr is actually a leading
cause of this type of overload, so you might actually be helping
contribute to the problem. Unfortunately many routers have hard-coded
rate-limits for icmp generation which can't be bumped, despite the fact
that we routinely hit them due to "ordinary" things like excessive mtr
use. I don't have a single router which hasn't bumped the rate limits by
pretty significant amounts, and most are dropping through the day under 
normal traffic loads:

ras at arandomrouter> show pfe statistics ip icmp | match throttled 
  5195868898 throttled icmps

Take a look at a presentation I did at NANOG 45 for more details.

http://www.nanog.org/meetings/nanog45/presentations/Sunday/RAS_traceroute_N45.pdf

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)