<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><br></div>Dual homing won’t help you if your automation template will do „no router bgp X” and at this point session will terminate as suddenly advertisement will be withdrawn…<div><br></div><div>It won’t you either if the change triggers some obscure bug in your BGP stack.</div><div><br></div><div>I bet FB tested the change on smaller scale and everything was fine, and only then started to roll this over wider network and at that point „something” broke. Or some bug needed a moment to start cascading issues around the infra.<br><br><div dir="ltr">-- <div>./</div></div><div dir="ltr"><br><blockquote type="cite">On 4 Oct 2021, at 22:00, Michael Thomas <mike@mtcc.com> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p><br>
</p>
<div class="moz-cite-prefix">On 10/4/21 11:48 AM, Luke Guillory
wrote:<br>
</div>
<blockquote type="cite" cite="mid:BN7PR02MB52013A16495D933E58A49711BEAE9@BN7PR02MB5201.namprd02.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><br>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe
UI",sans-serif;color:#242424;background:white">I
believe the original change was 'automatic' (as in
configuration done via a web interface). However, now that
connection to the outside world is down, remote access to
those tools don't exist anymore, so the emergency procedure
is to gain physical access to the peering routers and do all
the configuration locally.<o:p></o:p></span></p>
</div>
</blockquote>
<p>Assuming that this is what actually happened, what should fb have
done different (beyond the obvious of not screwing up the
immediate issue)? This seems like it's a single point of failure.
Should all of the BGP speakers have been dual homed or something
like that? Or should they not have been mixing ops and production
networks? Sorry if this sounds dumb.<br>
</p>
<p>Mike<br>
</p>
</div></blockquote></div></body></html>