Phantom packet loss is being shown when using pathping in connection with asynchronous routing - although there is no real loss.

Tue Jun 6 18:30:46 UTC 2006

"Gunther Stammwitz" <gstammw at gmx.net> wrote:
> I have customers who are complaining about packet loss and they are
> providing me with MTRs and pathpings (that's some sort of traceroute that
> pings every hop it sees several times - comes with windows xp) that show 
> the
> loss starting at my routers and ending at their server (=the last hop). 
> All
> users are coming from a (dialup-)network where the way from them to our
> servers are going via a carrier different than the carrier we are using to
> route the traffic back to the dial user.
> The interesting thing is that there is no loss at all when the users 
> either
> use a ping instead of this pathping/mtr-stuff or when I perform a ping or
> even an mtr on my server in direction of the dialup customer.
>
> The nasty thing is that there is de facto NO LOSS on the line but the 
> users
> is seeing some sort of phantom loss.
>
> The problem immediately disappears when I change to way back to the same
> carrier as the way to us so that we have synchronous routing again.
>
> My assumption is that pathping and mtr somehow get irritated by the icmp
> messages due to a wrong timing or something like that. Any ideas?

Try varying the mtr interval, such as "-i .1" (must be root for <1).  Does 
the packetloss significantly increase with this faster mtr?  Try slower "-i 
10".  Does the packetloss significantly decrease or go away?

If the answer to both above questions is yes, then I would suspect ICMP rate 
limiting.

You could also try varying the speed of ping.  Windows is pretty limited, 
but on unix you can do things like .1 second intervals ("-i .1" as root). 
Does a faster ping trigger this apparent loss?  If so, ICMP rate limiting.

The only part that I don't get is that you can mtr to him without 
packetloss.  Although the path in-between may be different, the final hop 
packetloss should exactly equal what he sees when mtring you.  A round-trip 
is a round-trip, and results should be identical regardless of who 
originates.  I can't think of any way this would be different unless echo 
and echo-reply were being rate limited independently.

My home ISP (apartment ethernet "t1" service, which is actually multiple 
T3s) has a Packeteer or something along that line.  If I use ping, 
everything is fine since it goes so slow.  If I use MTR, it works fine for 
the first few seconds then sees >90% packetloss on all hops from then on 
once the rate limiter burst bucket runs dry.  Of course, TCP still sees no 
packetloss even when mtr is seeing this heavy rate limited loss... 