CenturyLink RCA?

Mike Hammett nanog at ics-il.net
Sun Dec 30 15:42:22 UTC 2018


It's technical enough so that laypeople immediately lose interest, yet completely useless to anyone that works with this stuff. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

----- Original Message -----

From: "Saku Ytti" <saku at ytti.fi> 
To: "nanog list" <nanog at nanog.org> 
Sent: Sunday, December 30, 2018 7:42:49 AM 
Subject: CenturyLink RCA? 

Apologies for the URL, I do not know official source and I do not 
share the URLs sentiment. 
https://fuckingcenturylink.com/ 

Can someone translate this to IP engineer? What did actually happen? 
>From my own history, I rarely recognise the problem I fixed from 
reading the public RCA. I hope CenturyLink will do better. 

Best guess so far that I've heard is 

a) CenturyLink runs global L2 DCN/OOB 
b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU, 
I've had this failure mode) 
c) DCN had direct access to control-plane, and L2 congested 
control-plane resources causing it to deprovision waves 

Now of course this is entirely speculation, but intended to show what 
type of explanation is acceptable and can be used to fix things. 
Hopefully CenturyLink does come out with IP-engineering readable 
explanation, so that we may use it as leverage to support work in our 
own domains to remove such risks. 

a) do not run L2 DCN/OOB 
b) do not connect MGMT ETH (it is unprotected access to control-plane, 
it cannot be protected by CoPP/lo0 filter/LPTS ec) 
c) do add in your RFP scoring item for proper OOB port (Like Cisco CMP) 
d) do fail optical network up 

-- 
++ytti 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20181230/ce5ce260/attachment.html>


More information about the NANOG mailing list