Facebook post-mortems...

Tue Oct 5 16:44:05 UTC 2021

Mark Tinka wrote:
>
> So I'm not worried about DNS stability when split across multiple 
> physical entities.
>
> I'm talking about the actual services being hosted on a single network 
> that goes bye-bye like what we saw yesterday.
>
> All the DNS resolution means diddly, even if it tells us that DNS is 
> not the issue.
>
> Mark.
>
>
You could put up a temp page or two. Like, the internet is not down, we 
are just having a bad day. Bear with us for a bit. Go outside and enjoy 
nature for the next few hours.

But more importantly, internal infrastructure domains, containing router 
names, bootstraps, tools, utilities, physical access control, config 
repositories, network documentations, oob-network names (who remembers 
those?) , oob-email, oob communications (messenger, conferences, voip), 
etc..

Doesnt even have to be globally registered. External DNS server in the 
resolver list of all tech laptops slaving the zone.

Rapid response requires certain amenities, or as we can see, your 
talking about hours just getting started.

Also, the oob-network needs to be used regularly or it will be 
essentially unusable when actually needed, due to bit rot (accumulation 
of unnoticed and unresolved issues) and lack of mind muscle memory.

It should be standard practice to deploy all new equipment from the 
oob-network servicing it. Install things how you want to be able to 
repair them.

Joe