massive facebook outage presently

Tue Oct 5 04:31:26 UTC 2021

On 10/4/21 20:48, Luke Guillory wrote:

> I believe the original change was 'automatic' (as in configuration 
> done via a web interface). However, now that connection to the outside 
> world is down, remote access to those tools don't exist anymore, so 
> the emergency procedure is to gain physical access to the peering 
> routers and do all the configuration locally.
>

Q: What is automation?
A: Breaking the network at scale.

I suppose we shall know more in the coming days, but to me, it smells 
like a wide-scale automatic update roll-out that didn't go to plan. 
Isn't the first time a major content provider has suffered this; likely 
won't be the last.

In the end, the best thing for all of us is that it is a teaching 
moment, and we move the needle forward.

Mark.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20211005/837610fb/attachment.html>