<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p style="margin:0in;background:white"><span style="color:black">From what I believe was a FB employee on Reddit, account now deleted it seems.
<br>
<br>
<br>
</span><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#1A1A1B">As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has
gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).<o:p></o:p></span></p>
<p style="margin:0in;background:white;-webkit-font-smoothing: antialiased;box-sizing: border-box;font-variant-ligatures: normal;font-variant-caps: normal;orphans: 2;text-align:start;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424"><o:p> </o:p></span></p>
<p style="margin:0in;background:white;-webkit-font-smoothing: antialiased;box-sizing: border-box;font-variant-ligatures: normal;font-variant-caps: normal;orphans: 2;text-align:start;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424">There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually
authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.<o:p></o:p></span></p>
<p style="margin:0in;background:white;-webkit-font-smoothing: antialiased;box-sizing: border-box;font-variant-ligatures: normal;font-variant-caps: normal;orphans: 2;text-align:start;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424"><o:p> </o:p></span></p>
<p style="margin:0in;background:white;-webkit-font-smoothing: antialiased;box-sizing: border-box;font-variant-ligatures: normal;font-variant-caps: normal;orphans: 2;text-align:start;widows: 2;-webkit-text-stroke-width: 0px;text-decoration-style: initial;text-decoration-color: initial;word-spacing:0px">
<span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424">Part of this is also due to lower staffing in data centers due to pandemic measures.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white">I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world
is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://twitter.com/jgrahamc/status/1445068309288951820" target="_blank" title="https://twitter.com/jgrahamc/status/1445068309288951820"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#6264A7;background:white;text-decoration:none">https://twitter.com/jgrahamc/status/1445068309288951820</span></a><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white"> "About
five minutes before Facebook's DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook's ASN."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#242424;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> NANOG <nanog-bounces+lguillory=reservetele.com@nanog.org>
<b>On Behalf Of </b>Baldur Norddahl<br>
<b>Sent:</b> Monday, October 4, 2021 1:41 PM<br>
<b>To:</b> NANOG <nanog@nanog.org><br>
<b>Subject:</b> Re: massive facebook outage presently<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:12.0pt;font-family:"Courier New";color:black">*External Email: Use Caution*</span></b>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">I got a mail that Facebook was leaving NLIX. Maybe someone botched the script so they took down all BGP sessions instead of just NLIX and now they can't access the equipment to put it back... :-)
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">man. 4. okt. 2021 20.31 skrev Billy Croan <<a href="mailto:BCroan@unrealservers.net">BCroan@unrealservers.net</a>>:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">I know what this is..... They forgot to update the credit card on their godaddy account and the domain lapsed. I guess it will be
<a href="https://link.edgepilot.com/s/7bad5051/Di9CwLEB1E6iB_KlhyWtZA?u=http://facebook.info/" target="_blank">
facebook.info</a> when they get it back online. The post mortem should be an interesting read.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Oct 4, 2021 at 11:46 AM Jason Kuehl <<a href="mailto:jason.w.kuehl@gmail.com" target="_blank">jason.w.kuehl@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Looks like they run there own nameservers and I see the soa records are even missing.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Oct 4, 2021, 12:23 PM Mel Beckman <<a href="mailto:mel@beckman.org" target="_blank">mel@beckman.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Here’s a screenshot: <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p></o:p></p>
<div>
<p class="MsoNormal"> -mel beckman<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">On Oct 4, 2021, at 9:06 AM, Eric Kuhnke <<a href="mailto:eric.kuhnke@gmail.com" target="_blank">eric.kuhnke@gmail.com</a>> wrote:<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><a href="https://link.edgepilot.com/s/3926b9ff/bTkszib6zUmYbE_rZxhltQ?u=https://downdetector.com/status/facebook/" target="_blank">https://link.edgepilot.com/s/3926b9ff/bTkszib6zUmYbE_rZxhltQ?u=https://downdetector.com/status/facebook/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Normally not worth mentioning random $service having an outage here, but this will undoubtedly generate a large volume of customer service calls.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Appears to be failure in DNS resolution.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br>
Links contained in this email have been replaced. If you click on a link in the email above, the link will be analyzed for known threats. If a known threat is found, you will not be able to proceed to the destination. If suspicious content is detected, you
will see a warning.<o:p></o:p></p>
</div>
</div>
</body>
</html>