Level3 Internet service, out of order packets causing issues

Tue Jan 17 19:12:26 UTC 2017

Hi,

I have 1G Level3 ethernet dedicated internet service as one of my ISP's at my company based in the Los Angeles (Inland Empire) area. After seeing strange application behavior while using this circuit, I failed it out of service and have been troubleshooting it with a directly connected machine (publically addressed, no firewall, nothing between this machine and our Level3 router). I have taken several packet captures while accessing various sites and have noticed large numbers of out of order packets which are wreaking havoc with TCP connections and other traffic. In my experience, per-packet load balancing across various different links can cause this issue. I do not see this behavior with my other ISP's. I have had several tickets opened with Level3 but have had no success. Any help here? Anyone out there seen this and have any contacts that may be able to help?

FYI - we own our own public IP space and advertise via BGP to Level3. Currently I am using a dedicated /24 of our space advertised to Level3 only to ensure that the return path is through Level3 and not another ISP. Also, everything is single linked from a layer 2 and 3 perspective from the router to the test machine to ensure that the cause of any out of order packets is not on our end.

Thanks,

--
Mark Wicker | Senior Network Engineer
Esri | 380 New York St | Redlands, CA  92354 | USA
T 909 793 2853 x2741 | mwicker at esri.com<mailto:mwicker at esri.com> | esri.com<http://www.esri.com/>