Level3 Internet service, out of order packets causing issues

Wed Jan 18 02:56:51 UTC 2017

Hi Mark,
I'm going to throw out a guess here.  By any chance, is the first octet of
your router's MAC address a 4 or a 6?
In general, modern routers do not load balance per-packet, which is what
caused out-of-order issues in days gone.  Load balancing is usually done
based on a hash of the source and/or destination IP of the packet, MPLS
label, or Ethernet (on a switched interface).  The most common cause for
actual unordering of packets/frames in a modern service provider network,
in my experience, is actually this hashing mechanism.  Many vendor's
hashing implementations assume, based on position in the frame, that a
frame with a MAC address beginning with 4 or 6 is an IPv4 or IPv6 frame,
not an MPLS frame.  This can result in out of order packets.  The most
common fix is control word being applied on a pseudowire (assuming you are
being carried across a pseudowire in the SP network), but if this *is* what
is occurring, you could also resolve the issue by changing your MAC address.

On Tue, Jan 17, 2017 at 2:12 PM, Mark Wicker <MWicker at esri.com> wrote:

> Hi,
>
> I have 1G Level3 ethernet dedicated internet service as one of my ISP's at
> my company based in the Los Angeles (Inland Empire) area. After seeing
> strange application behavior while using this circuit, I failed it out of
> service and have been troubleshooting it with a directly connected machine
> (publically addressed, no firewall, nothing between this machine and our
> Level3 router). I have taken several packet captures while accessing
> various sites and have noticed large numbers of out of order packets which
> are wreaking havoc with TCP connections and other traffic. In my
> experience, per-packet load balancing across various different links can
> cause this issue. I do not see this behavior with my other ISP's. I have
> had several tickets opened with Level3 but have had no success. Any help
> here? Anyone out there seen this and have any contacts that may be able to
> help?
>
>
> FYI - we own our own public IP space and advertise via BGP to Level3.
> Currently I am using a dedicated /24 of our space advertised to Level3 only
> to ensure that the return path is through Level3 and not another ISP. Also,
> everything is single linked from a layer 2 and 3 perspective from the
> router to the test machine to ensure that the cause of any out of order
> packets is not on our end.
>
>
> Thanks,
>
>
> --
> Mark Wicker | Senior Network Engineer
> Esri | 380 New York St | Redlands, CA  92354 | USA
> T 909 793 2853 x2741 | mwicker at esri.com<mailto:mwicker at esri.com> |
> esri.com<http://www.esri.com/>
>
>