Level(3) ex-twtelecom midwest packet loss (4323)

Fri Aug 28 15:33:54 UTC 2015

I'll just leave this here....

https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/

On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady <jason at unlimitednet.us> wrote:
> Mike, I would take it to mean someone screwed something up and they don't
> want to admit to it.  :-)  That's just a guess.
>
> --
>
> Jason Canady
> Unlimited Net, LLC
> Responsive, Reliable, Secure
>
> www.unlimitednet.us
> jason at unlimitednet.us
> twitter: @unlimitednet
>
>
> On 8/28/15 12:00 AM, Mike Hammett wrote:
>>
>> 08/28/2015 3:08 AM GMT
>> Event Conclusion Summary
>>
>> Start: August 27, 2015 13:20 GMT
>> Stop: August 28, 2015 00:00 GMT
>>
>> Root Cause: A protocol issue impacted IP services in multiple markets.
>> Fix Action: Adjustments were made to clear the errors.
>>
>> Summary:
>> The IP NOC began investigating the root cause with Tier III Technical
>> Support. It was reported that the issue was causing packet loss for
>> customers. Operations Engineering teams were engaged, and Field Services
>> were dispatched to a site in Chicago, IL to assist with investigations.
>> Troubleshooting identified a protocol issue, and Operations Engineering
>> worked with Tier III Technical Support to perform adjustments on the links.
>> It was confirmed that the errors cleared. The traffic load was also lowered
>> on cards in Chicago to alleviate any further issues. Should any additional
>> impact be experienced, please contact the Level 3 Technical Service Center.
>>
>> What the hell is a "protocol issue"?
>>
>> I'm not an idiot, you can tell me specifically what happened...
>>
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> ----- Original Message -----
>>
>> From: "Ryan Gelobter" <ryan.g at atwgpc.net>
>> To: "Mel Beckman" <mel at beckman.org>
>> Cc: "<nanog at nanog.org>" <nanog at nanog.org>
>> Sent: Thursday, August 27, 2015 3:14:59 PM
>> Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323)
>>
>> If you have access to the Level3 portal you should see ticket #9639047
>> under Network Events now.
>>
>> Event Summary:IP Network Event ~ Multiple Markets
>>
>> 08/27/2015 8:05 PM GMT
>>
>> Level 3 Tier III and Operations Engineering teams have identified Internet
>> Protocols dropping, affecting customer services. Restoration efforts are
>> in progress, but an estimated time of restoral is not available at this
>> time.
>>
>> 08/27/2015 6:36 PM GMT
>>
>> IP and Transport Tier III, Operations Engineering and Field Services
>> continue collaboratively working the issue.
>>
>>
>> 08/27/2015 4:59 PM GMT
>>
>> Operations Engineering is engaged and Field Services is on site in
>> Chicago,
>> IL investigating the issue.
>>
>>
>> 08/27/2015 4:38 PM GMT
>>
>> The engineers are currently migrating traffic in efforts of restoring
>> services while troubleshooting continues. Field Services is being
>> dispatched to a Chicago, IL site to assist.
>>
>> 08/27/2015 4:21 PM GMT
>>
>> IP services are affected across multiple markets and the root cause is
>> currently under investigation. The IP NOC and IP and Transport Tier III
>> are
>> actively troubleshooting and working to isolate the cause. The engineers
>> have detected peering issues which are resulting in packet loss for
>> customers. Please be advised that updates will be provided at minimum of
>> hourly unless otherwise noted.
>>
>