<div dir="auto">If there isn't an undernetwork capable of being backdoored with the proper keys (I'd be shocked if there isn't - the big players have very good infra and DR people), I suspect there will be one soonish.<div dir="auto"><br></div><div dir="auto">It doesnt do much good to have DR plans and keys otherwise if you can't even get to the locks without getting on a plane.</div><div dir="auto"><br></div><div dir="auto">Regardless of how people may feel about the company, I just feel bad for their sres right now and am drinking one in their honor. I just want to know what an October meltdown gets called in the pm.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 4, 2021, 15:24 Baldur Norddahl <<a href="mailto:baldur.norddahl@gmail.com">baldur.norddahl@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Not in such a primitive fashion no. But they could definitely have a secondary network that will continue to work even if something goes wrong with the primary.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 4 Oct 2021 at 22:16, PJ Capelli <<a href="mailto:pjcapelli@pm.me" target="_blank" rel="noreferrer">pjcapelli@pm.me</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Seems unlikely that FB internal controls would allow such a backdoor ...</div><div><br></div><div><div><div>"Never to get lost, is not living" - Rebecca Solnit<br></div></div><div><br></div><div>Sent with <a href="https://protonmail.com/" target="_blank" rel="noreferrer">ProtonMail</a> Secure Email.</div></div><div><br></div><div>
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐<br>
On Monday, October 4th, 2021 at 4:12 PM, Baldur Norddahl <<a href="mailto:baldur.norddahl@gmail.com" target="_blank" rel="noreferrer">baldur.norddahl@gmail.com</a>> wrote:<br>
<blockquote type="cite">
<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Mon, 4 Oct 2021 at 21:58, Michael Thomas <<a href="mailto:mike@mtcc.com" rel="noreferrer nofollow noopener noreferrer" target="_blank">mike@mtcc.com</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
<div>
<p><br>
</p>
<div>On 10/4/21 11:48 AM, Luke Guillory
wrote:<br>
</div>
<blockquote type="cite">
<div><br>
<p><span>I
believe the original change was 'automatic' (as in
configuration done via a web interface). However, now that
connection to the outside world is down, remote access to
those tools don't exist anymore, so the emergency procedure
is to gain physical access to the peering routers and do all
the configuration locally.<u></u><u></u></span></p>
</div>
</blockquote>
<p>Assuming that this is what actually happened, what should fb have
done different (beyond the obvious of not screwing up the
immediate issue)? This seems like it's a single point of failure.
Should all of the BGP speakers have been dual homed or something
like that? Or should they not have been mixing ops and production
networks? Sorry if this sounds dumb.</p></div></blockquote><div><br></div><div>Facebook is a huge network. It is doubtful that what is going on is this simple. So I will make no guesses to what Facebook is or should be doing.</div><div><br></div><div>However the traditional way for us small timers is to have a backdoor using someone else's network. Nowadays this could be a simple 4/5G router with a VPN, to a terminal server that allows the operator to configure the equipment through the monitor port even when the config is completely destroyed.</div><div><br></div><div>Regards,</div><div><br></div><div>Baldur</div><div><br></div><div><br></div><div> </div></div></div>
</blockquote><br>
</div></blockquote></div></div>
</blockquote></div>