Level(3) ex-twtelecom midwest packet loss (4323)

Fri Aug 28 19:44:20 UTC 2015

Randy,

Frame Check Sequence errors are not a problem if you are capturing your packets from an endpoint system such as a PC participating in communications. Modern network cards do checksum offloading, so the FCS is calculated and applied by the NIC instead of by the OS. If you're capturing on one of the endpoints, Wireshark will see frames transmitted by that endpoint machine before the NIC inserts a correct FCS.

If communication succeeds (e.g., you see replies to pings), then there is not really an FCS error. If the FCS really was incorrect, the NIC would have discarded them, and you wouldn't see replies.

It's best to capture directly from the wire (for example, from a switch span port or a network tap) rather than an endpoint involved in the communication.

-mel

On Aug 28, 2015, at 11:49 AM, Randy <nanog at afxr.net<mailto:nanog at afxr.net>> wrote:

On 8/27/2015 11:00 PM, Mike Hammett wrote:
08/28/2015 3:08 AM GMT
Event Conclusion Summary

Start: August 27, 2015 13:20 GMT
Stop: August 28, 2015 00:00 GMT

Root Cause: A protocol issue impacted IP services in multiple markets.
Fix Action: Adjustments were made to clear the errors.

Summary:
The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center.

What the hell is a "protocol issue"?

I'm not an idiot, you can tell me specifically what happened...
I was having major issues yesterday. It wasn't with all traffic, and its seemed to have cleared up now.
I did do packet inspections, and I kept getting ethernet frame check errors from certain sources.
vast majority of packets did not have this extra data appended to the tcp portion of packet, and I think that is likely the reason for the issues, since those packets are supposed to be dropped if there is a checksum error. I'm by no means an expert, just voicing what I was seeing.

-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

----- Original Message -----

From: "Ryan Gelobter" <ryan.g at atwgpc.net<mailto:ryan.g at atwgpc.net>>
To: "Mel Beckman" <mel at beckman.org<mailto:mel at beckman.org>>
Cc: "<nanog at nanog.org<mailto:nanog at nanog.org>>" <nanog at nanog.org<mailto:nanog at nanog.org>>
Sent: Thursday, August 27, 2015 3:14:59 PM
Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323)

If you have access to the Level3 portal you should see ticket #9639047
under Network Events now.

Event Summary:IP Network Event ~ Multiple Markets

08/27/2015 8:05 PM GMT

Level 3 Tier III and Operations Engineering teams have identified Internet
Protocols dropping, affecting customer services. Restoration efforts are
in progress, but an estimated time of restoral is not available at this
time.

08/27/2015 6:36 PM GMT

IP and Transport Tier III, Operations Engineering and Field Services
continue collaboratively working the issue.

08/27/2015 4:59 PM GMT

Operations Engineering is engaged and Field Services is on site in Chicago,
IL investigating the issue.

08/27/2015 4:38 PM GMT

The engineers are currently migrating traffic in efforts of restoring
services while troubleshooting continues. Field Services is being
dispatched to a Chicago, IL site to assist.

08/27/2015 4:21 PM GMT

IP services are affected across multiple markets and the root cause is
currently under investigation. The IP NOC and IP and Transport Tier III are
actively troubleshooting and working to isolate the cause. The engineers
have detected peering issues which are resulting in packet loss for
customers. Please be advised that updates will be provided at minimum of
hourly unless otherwise noted.