massive facebook outage presently

Mon Oct 4 20:23:58 UTC 2021

Not in such a primitive fashion no. But they could definitely have a
secondary network that will continue to work even if something goes wrong
with the primary.

On Mon, 4 Oct 2021 at 22:16, PJ Capelli <pjcapelli at pm.me> wrote:

> Seems unlikely that FB internal controls would allow such a backdoor ...
>
> "Never to get lost, is not living" - Rebecca Solnit
>
> Sent with ProtonMail <https://protonmail.com/> Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <
> baldur.norddahl at gmail.com> wrote:
>
>
>
> On Mon, 4 Oct 2021 at 21:58, Michael Thomas <mike at mtcc.com> wrote:
>
>>
>> On 10/4/21 11:48 AM, Luke Guillory wrote:
>>
>>
>> I believe the original change was 'automatic' (as in configuration done
>> via a web interface). However, now that connection to the outside world is
>> down, remote access to those tools don't exist anymore, so the emergency
>> procedure is to gain physical access to the peering routers and do all the
>> configuration locally.
>>
>> Assuming that this is what actually happened, what should fb have done
>> different (beyond the obvious of not screwing up the immediate issue)? This
>> seems like it's a single point of failure. Should all of the BGP speakers
>> have been dual homed or something like that? Or should they not have been
>> mixing ops and production networks? Sorry if this sounds dumb.
>>
>
> Facebook is a huge network. It is doubtful that what is going on is this
> simple. So I will make no guesses to what Facebook is or should be doing.
>
> However the traditional way for us small timers is to have a backdoor
> using someone else's network. Nowadays this could be a simple 4/5G router
> with a VPN, to a terminal server that allows the operator to configure the
> equipment through the monitor port even when the config is completely
> destroyed.
>
> Regards,
>
> Baldur
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20211004/e5ba6672/attachment.html>