<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 10/4/21 11:48 AM, Luke Guillory
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BN7PR02MB52013A16495D933E58A49711BEAE9@BN7PR02MB5201.namprd02.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><br>
<p class="MsoNormal"><span
style="font-size:10.5pt;font-family:"Segoe
UI",sans-serif;color:#242424;background:white">I
believe the original change was 'automatic' (as in
configuration done via a web interface). However, now that
connection to the outside world is down, remote access to
those tools don't exist anymore, so the emergency procedure
is to gain physical access to the peering routers and do all
the configuration locally.<o:p></o:p></span></p>
</div>
</blockquote>
<p>Assuming that this is what actually happened, what should fb have
done different (beyond the obvious of not screwing up the
immediate issue)? This seems like it's a single point of failure.
Should all of the BGP speakers have been dual homed or something
like that? Or should they not have been mixing ops and production
networks? Sorry if this sounds dumb.<br>
</p>
<p>Mike<br>
</p>
</body>
</html>