Level(3) ex-twtelecom midwest packet loss (4323)

Fri Aug 28 16:06:44 UTC 2015

Personally I thought it was funny

I don't think it was meant to come across as offensive

On 8/28/2015 11:45 AM, Mel Beckman wrote:
> Blake,
>
> There's no call to be blatantly offensive like that.
>
> -mel
>
>> On Aug 28, 2015, at 8:35 AM, Blake Dunlap <ikiris at gmail.com> wrote:
>>
>> I'll just leave this here....
>>
>> https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/
>>
>>> On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady <jason at unlimitednet.us> wrote:
>>> Mike, I would take it to mean someone screwed something up and they don't
>>> want to admit to it.  :-)  That's just a guess.
>>>
>>> --
>>>
>>> Jason Canady
>>> Unlimited Net, LLC
>>> Responsive, Reliable, Secure
>>>
>>> www.unlimitednet.us
>>> jason at unlimitednet.us
>>> twitter: @unlimitednet
>>>
>>>
>>>> On 8/28/15 12:00 AM, Mike Hammett wrote:
>>>>
>>>> 08/28/2015 3:08 AM GMT
>>>> Event Conclusion Summary
>>>>
>>>> Start: August 27, 2015 13:20 GMT
>>>> Stop: August 28, 2015 00:00 GMT
>>>>
>>>> Root Cause: A protocol issue impacted IP services in multiple markets.
>>>> Fix Action: Adjustments were made to clear the errors.
>>>>
>>>> Summary:
>>>> The IP NOC began investigating the root cause with Tier III Technical
>>>> Support. It was reported that the issue was causing packet loss for
>>>> customers. Operations Engineering teams were engaged, and Field Services
>>>> were dispatched to a site in Chicago, IL to assist with investigations.
>>>> Troubleshooting identified a protocol issue, and Operations Engineering
>>>> worked with Tier III Technical Support to perform adjustments on the links.
>>>> It was confirmed that the errors cleared. The traffic load was also lowered
>>>> on cards in Chicago to alleviate any further issues. Should any additional
>>>> impact be experienced, please contact the Level 3 Technical Service Center.
>>>>
>>>> What the hell is a "protocol issue"?
>>>>
>>>> I'm not an idiot, you can tell me specifically what happened...
>>>>
>>>>
>>>>
>>>>
>>>> -----
>>>> Mike Hammett
>>>> Intelligent Computing Solutions
>>>> http://www.ics-il.com
>>>>
>>>> ----- Original Message -----
>>>>
>>>> From: "Ryan Gelobter" <ryan.g at atwgpc.net>
>>>> To: "Mel Beckman" <mel at beckman.org>
>>>> Cc: "<nanog at nanog.org>" <nanog at nanog.org>
>>>> Sent: Thursday, August 27, 2015 3:14:59 PM
>>>> Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323)
>>>>
>>>> If you have access to the Level3 portal you should see ticket #9639047
>>>> under Network Events now.
>>>>
>>>> Event Summary:IP Network Event ~ Multiple Markets
>>>>
>>>> 08/27/2015 8:05 PM GMT
>>>>
>>>> Level 3 Tier III and Operations Engineering teams have identified Internet
>>>> Protocols dropping, affecting customer services. Restoration efforts are
>>>> in progress, but an estimated time of restoral is not available at this
>>>> time.
>>>>
>>>> 08/27/2015 6:36 PM GMT
>>>>
>>>> IP and Transport Tier III, Operations Engineering and Field Services
>>>> continue collaboratively working the issue.
>>>>
>>>>
>>>> 08/27/2015 4:59 PM GMT
>>>>
>>>> Operations Engineering is engaged and Field Services is on site in
>>>> Chicago,
>>>> IL investigating the issue.
>>>>
>>>>
>>>> 08/27/2015 4:38 PM GMT
>>>>
>>>> The engineers are currently migrating traffic in efforts of restoring
>>>> services while troubleshooting continues. Field Services is being
>>>> dispatched to a Chicago, IL site to assist.
>>>>
>>>> 08/27/2015 4:21 PM GMT
>>>>
>>>> IP services are affected across multiple markets and the root cause is
>>>> currently under investigation. The IP NOC and IP and Transport Tier III
>>>> are
>>>> actively troubleshooting and working to isolate the cause. The engineers
>>>> have detected peering issues which are resulting in packet loss for
>>>> customers. Please be advised that updates will be provided at minimum of
>>>> hourly unless otherwise noted.