Level3 Internet service, out of order packets causing issues

Mark Wicker MWicker at esri.com
Wed Jan 18 18:57:37 UTC 2017


Hi Jason,

I think that this was it. The first octet of our router’s Level3 facing interface MAC is a 6. Changed it to 0 and it does look much better now. Thanks!


--
Mark Wicker | Senior Network Engineer
Esri | 380 New York St | Redlands, CA  92354 | USA
T 909 793 2853 x2741 | mwicker at esri.com<mailto:mwicker at esri.com> | esri.com<http://www.esri.com/>

From: Jason Rokeach [mailto:jason at rokeach.net]
Sent: Tuesday, January 17, 2017 6:57 PM
To: Mark Wicker <MWicker at esri.com>
Cc: nanog at nanog.org
Subject: Re: Level3 Internet service, out of order packets causing issues

Hi Mark,
I'm going to throw out a guess here.  By any chance, is the first octet of your router's MAC address a 4 or a 6?
In general, modern routers do not load balance per-packet, which is what caused out-of-order issues in days gone.  Load balancing is usually done based on a hash of the source and/or destination IP of the packet, MPLS label, or Ethernet (on a switched interface).  The most common cause for actual unordering of packets/frames in a modern service provider network, in my experience, is actually this hashing mechanism.  Many vendor's hashing implementations assume, based on position in the frame, that a frame with a MAC address beginning with 4 or 6 is an IPv4 or IPv6 frame, not an MPLS frame.  This can result in out of order packets.  The most common fix is control word being applied on a pseudowire (assuming you are being carried across a pseudowire in the SP network), but if this is what is occurring, you could also resolve the issue by changing your MAC address.


On Tue, Jan 17, 2017 at 2:12 PM, Mark Wicker <MWicker at esri.com<mailto:MWicker at esri.com>> wrote:
Hi,

I have 1G Level3 ethernet dedicated internet service as one of my ISP's at my company based in the Los Angeles (Inland Empire) area. After seeing strange application behavior while using this circuit, I failed it out of service and have been troubleshooting it with a directly connected machine (publically addressed, no firewall, nothing between this machine and our Level3 router). I have taken several packet captures while accessing various sites and have noticed large numbers of out of order packets which are wreaking havoc with TCP connections and other traffic. In my experience, per-packet load balancing across various different links can cause this issue. I do not see this behavior with my other ISP's. I have had several tickets opened with Level3 but have had no success. Any help here? Anyone out there seen this and have any contacts that may be able to help?


FYI - we own our own public IP space and advertise via BGP to Level3. Currently I am using a dedicated /24 of our space advertised to Level3 only to ensure that the return path is through Level3 and not another ISP. Also, everything is single linked from a layer 2 and 3 perspective from the router to the test machine to ensure that the cause of any out of order packets is not on our end.


Thanks,


--
Mark Wicker | Senior Network Engineer
Esri | 380 New York St | Redlands, CA  92354 | USA
T 909 793 2853 x2741 | mwicker at esri.com<mailto:mwicker at esri.com><mailto:mwicker at esri.com<mailto:mwicker at esri.com>> | esri.com<http://esri.com><http://www.esri.com/>



More information about the NANOG mailing list