CenturyLink RCA?

Mon Dec 31 15:26:52 UTC 2018

I agree 100%.  Now they need to figure out why bricking the management network stopped forwarding on the optical side.  > (Forgive my top posting, not on my desktop as I’m out of town)

Steven Naslund
Chicago IL
>
>Wild guess, based on my own experience as a NOC admin/head of operations at a large ISP - they have an automated deployment system for new firmware for a (mission critical) piece of backbone hardware.
>
>They may have tested said firmware on a chassis with cards that did not exactly match the hardware they had in actual deployment (ie: card was older hw revision in deployed hardware), and while it worked fine there, it proceeded >shit the bed in the production.
>
>Or, they missed a mandatory low level hardware firmware upgrade that has to be applied separately before the other main upgrade.
>
>Kinda picturing in my mind that they staged all the updates, set a timer, staggered reboot, and after the first hit the fan, they couldn’t stop the rest as it fell apart as each upgraded unit fell on its own sword on reboot.
>
>I’ve been bit by the ‘this card revision is not supported under this platform/release’ bug more often then I’d like to admit.
>
>And, yes, my eyes did start to get glossy and hazy the more I read their explanation as well.  It’s exactly the kind of useless post I’d write when I want to get (stupid) people off my back about a problem.