Facebook post-mortems...

Tue Oct 5 13:40:39 UTC 2021

On 10/5/21 15:04, Joe Greco wrote:

>
> You don't think at least 10,000 helpdesk requests about Facebook being
> down were sent yesterday?

That and Jane + Thando likely re-installing all their apps and 
iOS/Android on their phones, and rebooting them 300 times in the hopes 
that Facebook and WhatsApp would work.

Yes, total nightmare yesterday, but sure that 9,999 of the helpdesk 
tickets had nothing to do with DNS. They likely all were - "Your 
Internet is down, just fix it; we don't wanna know".

> There's something to be said for building these things to be resilient
> in a manner that isn't just convenient internally, but also externally
> to those people that network operators sometimes forget also support
> their network issues indirectly.

I don't disagree with you one bit. It's for that exact reason that we built:

     https://as37100.net/

... not for us, but specifically for other random network operators 
around the world whom we may never get to drink a crate of wine with.

I have to say that it has likely cut e-mails to our NOC as well as 
overall pain in half, if not more.

Mark.