is something weird going on with cox, level3, and/or cogent

Mon Feb 13 03:45:36 UTC 2017

It looks like one or more circuits are down, so you're seeing asymmetrical routing over congested paths in one direction. 

-mel via cell

> On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman at meetinghouse.net> wrote:
> 
> Hi Folks,
> 
> I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via Cox broadband.  The folks at Cox technical support are useless - all they say is "well you're on a DOCSIS 2 modem."  Meanwhile, everything I'm seeing is several hops upstream of the local segment - and all that Cox level2 tech support will say is "if there was a backbone problem, our backbone people would have dealt with it."
> 
> I'm having problems reaching both our own server, and sites like google, facebook, windows update.
> 
> Traceroutes to and from our server are illustrative - note that for most of the past week, the average ping time was 85msec. Now we're seeing this:
> 
> From 98.177.135.186 - the public IP address on Cox's local broadband service.
> To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint data center, near Boston)
> traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
> 1  10.128.128.1 (10.128.128.1)  199.893 ms  75.319 ms  27.295 ms
> 2  100.127.69.178 (100.127.69.178)  38.710 ms  40.075 ms  43.598 ms
> 3  72.215.229.22 (72.215.229.22)  39.674 ms  201.368 ms *
> 4  lag-157.bear2.phoenix1.level3.net (4.28.82.53)  686.499 ms 1837.141 ms  16.273 ms
> 5  ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  35.498 ms 964.377 ms *
> 6  ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  551.760 ms  525.014 ms
>    ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  2061.191 ms
> 7  be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129)  847.778 ms  87.601 ms  71.504 ms
> 8  be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1)  79.060 ms  225.647 ms
>    be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77)  60.306 ms
> 9  * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms  185.180 ms
> 10  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  61.208 ms
>    be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  386.149 ms  1278.868 ms
> 11  be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161)  384.136 ms
>    be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms  615.415 ms
> 12  be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  233.061 ms
>    be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69)  87.902 ms
>    be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  861.159 ms
> 13  be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221)  998.858 ms
>    be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  249.930 ms *
> 14  be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  768.461 ms
>    be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  136.772 ms
>    be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  288.225 ms
> 15  be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14)  271.736 ms  166.224 ms
>    be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42)  565.015 ms
> 16  te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms
>    te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms
>    te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms
> 17  te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82)  107.207 ms
>    te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78)  295.881 ms  185.453 ms
> 18  38.122.127.18 (38.122.127.18)  115.652 ms  461.168 ms  615.526 ms
> 19  207.154.0.57 (207.154.0.57)  1871.023 ms  1987.832 ms 2165.248 ms
> 20  server1.ntcorp.com (207.154.13.58)  587.560 ms  263.328 ms 333.542 ms
> 
> 
> Traceroute in the reverse direction:
> 1  207.154.13.47 (207.154.13.47)  0.000 ms  0.000 ms  0.000 ms
> 2  * * *
> 3  h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms  0.000 ms  0.000 ms
> 4  xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms  12.000 ms  12.000 ms
> 5  et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms  20.000 ms  20.000 ms
> 6  10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37)  72.001 ms  20.000 ms  16.000 ms
> 7  chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193)  20.000 ms 20.000 ms  20.000 ms
> 8  chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211)  72.001 ms  72.001 ms  72.001 ms
> 9  * * *
> 10  * * *
> 11  * * *
> <snip>
> 30  * * *
> <looks like a lot of the later hops don't respond to pings>
> 
> Two things jump out at me:
> 1. The rather large number of hops from cox to ntcorp - with high delays  from several nodes in both the level3 and cogent networks.
> 2. That there's a rather more direct path from the datacenter to cox, that shows up in the reverse direction.
> 
> Some kind of routing or peering issue, perhaps?  (And I also note the earlier string of messages regarding youtube streaming problems - that also seemed to involve cox and level3.
> 
> Thanks for any insight (and better, for any fixes!).
> 
> Miles Fidelman
> 
> 
> 
> 
> -- 
> In theory, there is no difference between theory and practice.
> In practice, there is.  .... Yogi Berra
> 