Consistent asymetric latency on monitoring?

Thu Oct 22 01:31:00 UTC 2009

Rick Ernst wrote:
> Although the implementation is Cisco-specific, this feels more appropriate
> for NANOG.
>
> We've started rolling out a state-wide monitoring system based on Cisco's
> "IP SLA" feature set.  Out of 5 sites deployed so far (different locations,
> different providers), we are consistently seeing one-way latency mirror the
> opposite direction. As source-destination latency goes up,
> destination-source latency goes down and vice versa.
>
> Myself and the monitoring team have ripped apart the OIDs, IP SLA
> configuration, and monitoring system.  We've also built an ad-hoc system to
> compare the results.  It's still consistent behavior.  It's not a true
> mirror; there is definitely variation between the data collection, but at
> the 10,000 foot level, there is an obvious and consistent mirror to the
> data.
>
> The network topology is independant service providers all providing backhaul
> to a local ethernet exchange.
>
> Has anybody seen this type of behavior? We are solidly convinced that we are
> using the proper OIDs and making the proper transformations of the data.
> The two remaining causes appear to be either "natural behavior of the links"
> and/or "artifact in the IP SLA mechanism".
>
> Any ideas?
>   

Having never used cisco's IP SLA (or even read about it), take this with 
a sack of salt.

I assume this product works by having a packet with a timestamp sent 
from the source to the destination where it is timestamped again and 
either sent back, or another packet is sent in the other direction.  The 
difference between the two timestamps gives you the latency in that 
direction.

Now, how are your clocks syncronised?  are they synchronised using NTP? 
or something better (GPS?)  If one of your clocks is drifting with 
respect to the other then you'll see this effect.  Does your clock drift 
because NTP is failing to keep the clock well syncronised when it's 
connection to it's parent NTP server is saturated?