Level(3) ex-twtelecom midwest packet loss (4323)

Fri Aug 28 17:04:35 UTC 2015

ill just leave this here.... :)

https://www.youtube.com/watch?v=0ilMx7k7mso

On Fri, Aug 28, 2015 at 11:33 AM, Blake Dunlap <ikiris at gmail.com> wrote:

> I'll just leave this here....
>
>
> https://honestnetworker.wordpress.com/2013/11/04/the-true-meaning-behind-most-rfos/
>
> On Fri, Aug 28, 2015 at 8:26 AM, Jason Canady <jason at unlimitednet.us>
> wrote:
> > Mike, I would take it to mean someone screwed something up and they don't
> > want to admit to it.  :-)  That's just a guess.
> >
> > --
> >
> > Jason Canady
> > Unlimited Net, LLC
> > Responsive, Reliable, Secure
> >
> > www.unlimitednet.us
> > jason at unlimitednet.us
> > twitter: @unlimitednet
> >
> >
> > On 8/28/15 12:00 AM, Mike Hammett wrote:
> >>
> >> 08/28/2015 3:08 AM GMT
> >> Event Conclusion Summary
> >>
> >> Start: August 27, 2015 13:20 GMT
> >> Stop: August 28, 2015 00:00 GMT
> >>
> >> Root Cause: A protocol issue impacted IP services in multiple markets.
> >> Fix Action: Adjustments were made to clear the errors.
> >>
> >> Summary:
> >> The IP NOC began investigating the root cause with Tier III Technical
> >> Support. It was reported that the issue was causing packet loss for
> >> customers. Operations Engineering teams were engaged, and Field Services
> >> were dispatched to a site in Chicago, IL to assist with investigations.
> >> Troubleshooting identified a protocol issue, and Operations Engineering
> >> worked with Tier III Technical Support to perform adjustments on the
> links.
> >> It was confirmed that the errors cleared. The traffic load was also
> lowered
> >> on cards in Chicago to alleviate any further issues. Should any
> additional
> >> impact be experienced, please contact the Level 3 Technical Service
> Center.
> >>
> >> What the hell is a "protocol issue"?
> >>
> >> I'm not an idiot, you can tell me specifically what happened...
> >>
> >>
> >>
> >>
> >> -----
> >> Mike Hammett
> >> Intelligent Computing Solutions
> >> http://www.ics-il.com
> >>
> >> ----- Original Message -----
> >>
> >> From: "Ryan Gelobter" <ryan.g at atwgpc.net>
> >> To: "Mel Beckman" <mel at beckman.org>
> >> Cc: "<nanog at nanog.org>" <nanog at nanog.org>
> >> Sent: Thursday, August 27, 2015 3:14:59 PM
> >> Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323)
> >>
> >> If you have access to the Level3 portal you should see ticket #9639047
> >> under Network Events now.
> >>
> >> Event Summary:IP Network Event ~ Multiple Markets
> >>
> >> 08/27/2015 8:05 PM GMT
> >>
> >> Level 3 Tier III and Operations Engineering teams have identified
> Internet
> >> Protocols dropping, affecting customer services. Restoration efforts are
> >> in progress, but an estimated time of restoral is not available at this
> >> time.
> >>
> >> 08/27/2015 6:36 PM GMT
> >>
> >> IP and Transport Tier III, Operations Engineering and Field Services
> >> continue collaboratively working the issue.
> >>
> >>
> >> 08/27/2015 4:59 PM GMT
> >>
> >> Operations Engineering is engaged and Field Services is on site in
> >> Chicago,
> >> IL investigating the issue.
> >>
> >>
> >> 08/27/2015 4:38 PM GMT
> >>
> >> The engineers are currently migrating traffic in efforts of restoring
> >> services while troubleshooting continues. Field Services is being
> >> dispatched to a Chicago, IL site to assist.
> >>
> >> 08/27/2015 4:21 PM GMT
> >>
> >> IP services are affected across multiple markets and the root cause is
> >> currently under investigation. The IP NOC and IP and Transport Tier III
> >> are
> >> actively troubleshooting and working to isolate the cause. The engineers
> >> have detected peering issues which are resulting in packet loss for
> >> customers. Please be advised that updates will be provided at minimum of
> >> hourly unless otherwise noted.
> >>
> >
>