<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-CA link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='mso-fareast-language:EN-US'>I agree to resolve non-routable address doesn’t bring you a working service.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>I thought a few networks were still reachable like their MX or some DRP networks.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Thanks for the update<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Jean<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> Tom Beecher <beecher@beecher.cc> <br><b>Sent:</b> October 5, 2021 8:33 AM<br><b>To:</b> Jean St-Laurent <jean@ddostest.me><br><b>Cc:</b> Jeff Tantsura <jefftant.ietf@gmail.com>; William Herrin <bill@herrin.us>; NANOG <nanog@nanog.org><br><b>Subject:</b> Re: Facebook post-mortems...<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><p class=MsoNormal>Maybe withdrawing those routes to their NS could have been mitigated by having NS in separate entities.<o:p></o:p></p></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Assuming they had such a thing in place , it would not have helped. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Facebook stopped announcing the vast majority of their IP space to the DFZ during this. So even they did have an offnet DNS server that could have provided answers to clients, those same clients probably wouldn't have been able to connect to the IPs returned anyways. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>If you are running your own auths like they are, you likely view your public network reachability as almost bulletproof and that it will never disappear. Which is probably true most of the time. Until yesterday happens and the 9's in your reliability percentage change to 7's. <o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Tue, Oct 5, 2021 at 8:10 AM Jean St-Laurent via NANOG <<a href="mailto:nanog@nanog.org">nanog@nanog.org</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><p class=MsoNormal style='margin-bottom:12.0pt'>Maybe withdrawing those routes to their NS could have been mitigated by having NS in separate entities.<br><br>Let's check how these big companies are spreading their NS's.<br><br>$ dig +short <a href="http://facebook.com" target="_blank">facebook.com</a> NS<br><a href="http://d.ns.facebook.com" target="_blank">d.ns.facebook.com</a>.<br><a href="http://b.ns.facebook.com" target="_blank">b.ns.facebook.com</a>.<br><a href="http://c.ns.facebook.com" target="_blank">c.ns.facebook.com</a>.<br><a href="http://a.ns.facebook.com" target="_blank">a.ns.facebook.com</a>.<br><br>$ dig +short <a href="http://google.com" target="_blank">google.com</a> NS<br><a href="http://ns1.google.com" target="_blank">ns1.google.com</a>.<br><a href="http://ns4.google.com" target="_blank">ns4.google.com</a>.<br><a href="http://ns2.google.com" target="_blank">ns2.google.com</a>.<br><a href="http://ns3.google.com" target="_blank">ns3.google.com</a>.<br><br>$ dig +short <a href="http://apple.com" target="_blank">apple.com</a> NS<br><a href="http://a.ns.apple.com" target="_blank">a.ns.apple.com</a>.<br><a href="http://b.ns.apple.com" target="_blank">b.ns.apple.com</a>.<br><a href="http://c.ns.apple.com" target="_blank">c.ns.apple.com</a>.<br><a href="http://d.ns.apple.com" target="_blank">d.ns.apple.com</a>.<br><br>$ dig +short <a href="http://amazon.com" target="_blank">amazon.com</a> NS<br><a href="http://ns4.p31.dynect.net" target="_blank">ns4.p31.dynect.net</a>.<br><a href="http://ns3.p31.dynect.net" target="_blank">ns3.p31.dynect.net</a>.<br><a href="http://ns1.p31.dynect.net" target="_blank">ns1.p31.dynect.net</a>.<br><a href="http://ns2.p31.dynect.net" target="_blank">ns2.p31.dynect.net</a>.<br><a href="http://pdns6.ultradns.co.uk" target="_blank">pdns6.ultradns.co.uk</a>.<br><a href="http://pdns1.ultradns.net" target="_blank">pdns1.ultradns.net</a>.<br><br>$ dig +short <a href="http://netflix.com" target="_blank">netflix.com</a> NS<br><a href="http://ns-1372.awsdns-43.org" target="_blank">ns-1372.awsdns-43.org</a>.<br><a href="http://ns-1984.awsdns-56.co.uk" target="_blank">ns-1984.awsdns-56.co.uk</a>.<br><a href="http://ns-659.awsdns-18.net" target="_blank">ns-659.awsdns-18.net</a>.<br><a href="http://ns-81.awsdns-10.com" target="_blank">ns-81.awsdns-10.com</a>.<br><br>Amnazon and Netflix seem to not keep their eggs in the same basket. From a first look, they seem more resilient than <a href="http://facebook.com" target="_blank">facebook.com</a>, <a href="http://google.com" target="_blank">google.com</a> and <a href="http://apple.com" target="_blank">apple.com</a><br><br>Jean<br><br>-----Original Message-----<br>From: NANOG <nanog-bounces+jean=<a href="mailto:ddostest.me@nanog.org" target="_blank">ddostest.me@nanog.org</a>> On Behalf Of Jeff Tantsura<br>Sent: October 5, 2021 2:18 AM<br>To: William Herrin <<a href="mailto:bill@herrin.us" target="_blank">bill@herrin.us</a>><br>Cc: <a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a><br>Subject: Re: Facebook post-mortems...<br><br><a href="http://129.134.30.0/23" target="_blank">129.134.30.0/23</a>, <a href="http://129.134.30.0/24" target="_blank">129.134.30.0/24</a>, <a href="http://129.134.31.0/24" target="_blank">129.134.31.0/24</a>. The specific routes covering all 4 nameservers (a-d) were withdrawn from all FB peering at approximately 15:40 UTC.<br><br>Cheers,<br>Jeff<br><br>> On Oct 4, 2021, at 22:45, William Herrin <<a href="mailto:bill@herrin.us" target="_blank">bill@herrin.us</a>> wrote:<br>> <br>> On Mon, Oct 4, 2021 at 6:15 PM Michael Thomas <<a href="mailto:mike@mtcc.com" target="_blank">mike@mtcc.com</a>> wrote:<br>>> They have a monkey patch subsystem. Lol.<br>> <br>> Yes, actually, they do. They use Chef extensively to configure <br>> operating systems. Chef is written in Ruby. Ruby has something called <br>> Monkey Patches. This is where at an arbitrary location in the code you <br>> re-open an object defined elsewhere and change its methods.<br>> <br>> Chef doesn't always do the right thing. You tell Chef to remove an RPM <br>> and it does. Even if it has to remove half the operating system to <br>> satisfy the dependencies. If you want it to do something reasonable, <br>> say throw an error because you didn't actually tell it to remove half <br>> the operating system, you have a choice: spin up a fork of chef with a <br>> couple patches to the chef-rpm interaction or just monkey-patch it in <br>> one of your chef recipes.<br>> <br>> Regards,<br>> Bill Herrin<br>> <br>> --<br>> William Herrin<br>> <a href="mailto:bill@herrin.us" target="_blank">bill@herrin.us</a><br>> <a href="https://bill.herrin.us/" target="_blank">https://bill.herrin.us/</a><o:p></o:p></p></blockquote></div></div></body></html>