is something weird going on with cox, level3, and/or cogent

Mon Feb 13 03:57:22 UTC 2017

Miles,

Have you tried trace routing through cellular data connections? The results you're seeing could be explained by congestion at the point of your modem, which I think is with the cox techs are implying.

-mel via cell

> On Feb 12, 2017, at 7:46 PM, Mel Beckman <mel at beckman.org> wrote:
> 
> It looks like one or more circuits are down, so you're seeing asymmetrical routing over congested paths in one direction. 
> 
> -mel via cell
> 
>> On Feb 12, 2017, at 7:14 PM, Miles Fidelman <mfidelman at meetinghouse.net> wrote:
>> 
>> Hi Folks,
>> 
>> I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via Cox broadband.  The folks at Cox technical support are useless - all they say is "well you're on a DOCSIS 2 modem."  Meanwhile, everything I'm seeing is several hops upstream of the local segment - and all that Cox level2 tech support will say is "if there was a backbone problem, our backbone people would have dealt with it."
>> 
>> I'm having problems reaching both our own server, and sites like google, facebook, windows update.
>> 
>> Traceroutes to and from our server are illustrative - note that for most of the past week, the average ping time was 85msec. Now we're seeing this:
>> 
>> From 98.177.135.186 - the public IP address on Cox's local broadband service.
>> To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint data center, near Boston)
>> traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
>> 1  10.128.128.1 (10.128.128.1)  199.893 ms  75.319 ms  27.295 ms
>> 2  100.127.69.178 (100.127.69.178)  38.710 ms  40.075 ms  43.598 ms
>> 3  72.215.229.22 (72.215.229.22)  39.674 ms  201.368 ms *
>> 4  lag-157.bear2.phoenix1.level3.net (4.28.82.53)  686.499 ms 1837.141 ms  16.273 ms
>> 5  ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  35.498 ms 964.377 ms *
>> 6  ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  551.760 ms  525.014 ms
>>   ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  2061.191 ms
>> 7  be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129)  847.778 ms  87.601 ms  71.504 ms
>> 8  be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1)  79.060 ms  225.647 ms
>>   be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77)  60.306 ms
>> 9  * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms  185.180 ms
>> 10  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  61.208 ms
>>   be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  386.149 ms  1278.868 ms
>> 11  be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161)  384.136 ms
>>   be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms  615.415 ms
>> 12  be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  233.061 ms
>>   be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69)  87.902 ms
>>   be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  861.159 ms
>> 13  be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221)  998.858 ms
>>   be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  249.930 ms *
>> 14  be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  768.461 ms
>>   be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  136.772 ms
>>   be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  288.225 ms
>> 15  be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14)  271.736 ms  166.224 ms
>>   be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42)  565.015 ms
>> 16  te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms
>>   te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms
>>   te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms
>> 17  te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82)  107.207 ms
>>   te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78)  295.881 ms  185.453 ms
>> 18  38.122.127.18 (38.122.127.18)  115.652 ms  461.168 ms  615.526 ms
>> 19  207.154.0.57 (207.154.0.57)  1871.023 ms  1987.832 ms 2165.248 ms
>> 20  server1.ntcorp.com (207.154.13.58)  587.560 ms  263.328 ms 333.542 ms
>> 
>> 
>> Traceroute in the reverse direction:
>> 1  207.154.13.47 (207.154.13.47)  0.000 ms  0.000 ms  0.000 ms
>> 2  * * *
>> 3  h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms  0.000 ms  0.000 ms
>> 4  xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms  12.000 ms  12.000 ms
>> 5  et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms  20.000 ms  20.000 ms
>> 6  10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37)  72.001 ms  20.000 ms  16.000 ms
>> 7  chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193)  20.000 ms 20.000 ms  20.000 ms
>> 8  chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211)  72.001 ms  72.001 ms  72.001 ms
>> 9  * * *
>> 10  * * *
>> 11  * * *
>> <snip>
>> 30  * * *
>> <looks like a lot of the later hops don't respond to pings>
>> 
>> Two things jump out at me:
>> 1. The rather large number of hops from cox to ntcorp - with high delays  from several nodes in both the level3 and cogent networks.
>> 2. That there's a rather more direct path from the datacenter to cox, that shows up in the reverse direction.
>> 
>> Some kind of routing or peering issue, perhaps?  (And I also note the earlier string of messages regarding youtube streaming problems - that also seemed to involve cox and level3.
>> 
>> Thanks for any insight (and better, for any fixes!).
>> 
>> Miles Fidelman
>> 
>> 
>> 
>> 
>> -- 
>> In theory, there is no difference between theory and practice.
>> In practice, there is.  .... Yogi Berra
>> 