massive facebook outage presently

Mon Oct 4 20:32:08 UTC 2021

If there isn't an undernetwork capable of being backdoored with the proper
keys (I'd be shocked if there isn't - the big players have very good infra
and DR people), I suspect there will be one soonish.

It doesnt do much good to have DR plans and keys otherwise if you can't
even get to the locks without getting on a plane.

Regardless of how people may feel about the company, I just feel bad for
their sres right now and am drinking one in their honor. I just want to
know what an October meltdown gets called in the pm.

On Mon, Oct 4, 2021, 15:24 Baldur Norddahl <baldur.norddahl at gmail.com>
wrote:

> Not in such a primitive fashion no. But they could definitely have a
> secondary network that will continue to work even if something goes wrong
> with the primary.
>
> On Mon, 4 Oct 2021 at 22:16, PJ Capelli <pjcapelli at pm.me> wrote:
>
>> Seems unlikely that FB internal controls would allow such a backdoor ...
>>
>> "Never to get lost, is not living" - Rebecca Solnit
>>
>> Sent with ProtonMail <https://protonmail.com/> Secure Email.
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <
>> baldur.norddahl at gmail.com> wrote:
>>
>>
>>
>> On Mon, 4 Oct 2021 at 21:58, Michael Thomas <mike at mtcc.com> wrote:
>>
>>>
>>> On 10/4/21 11:48 AM, Luke Guillory wrote:
>>>
>>>
>>> I believe the original change was 'automatic' (as in configuration done
>>> via a web interface). However, now that connection to the outside world is
>>> down, remote access to those tools don't exist anymore, so the emergency
>>> procedure is to gain physical access to the peering routers and do all the
>>> configuration locally.
>>>
>>> Assuming that this is what actually happened, what should fb have done
>>> different (beyond the obvious of not screwing up the immediate issue)? This
>>> seems like it's a single point of failure. Should all of the BGP speakers
>>> have been dual homed or something like that? Or should they not have been
>>> mixing ops and production networks? Sorry if this sounds dumb.
>>>
>>
>> Facebook is a huge network. It is doubtful that what is going on is this
>> simple. So I will make no guesses to what Facebook is or should be doing.
>>
>> However the traditional way for us small timers is to have a backdoor
>> using someone else's network. Nowadays this could be a simple 4/5G router
>> with a VPN, to a terminal server that allows the operator to configure the
>> equipment through the monitor port even when the config is completely
>> destroyed.
>>
>> Regards,
>>
>> Baldur
>>
>>
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20211004/124d4165/attachment-0001.html>