<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 10/5/21 3:09 PM, Andy Brezinsky
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:e737efc6-2b39-fa83-7aed-162792f16a16@mbrez.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>It's a few years old, but Facebook has talked a little bit
about their DNS infrastructure before. Here's a little clip
that talks about Cartographer: <a class="moz-txt-link-freetext"
href="https://youtu.be/bxhYNfFeVF4?t=2073"
moz-do-not-send="true">https://youtu.be/bxhYNfFeVF4?t=2073</a>
<br>
</p>
<p>From their outage report, it sounds like their authoritative
DNS servers withdraw their anycast announcements when they're
unhealthy. The health check from those servers must have relied
on something upstream. Maybe they couldn't talk to Cartographer
for a few minutes so they thought they might be isolated from
the rest of the network and they decided to withdraw their
routes instead of serving stale data. Makes sense when a single
node does it, not so much when the entire fleet thinks that
they're out on their own.<br>
</p>
<p>A performance issue in Cartographer (or whatever manages this
fleet these days) could have been the ticking time bomb that set
the whole thing in motion.<br>
</p>
</blockquote>
<p>Rereading it is said that their internal (?) backbone went down
so pulling the routes was arguably the right thing to do. Or at
least not flat out wrong. Taking out their nameserver subnets was
clearly a problem though, though a fix is probably tricky since
you clearly want to take down errant nameservers too. <br>
</p>
<p><br>
</p>
<p>Mike<br>
</p>
<br>
<blockquote type="cite"
cite="mid:e737efc6-2b39-fa83-7aed-162792f16a16@mbrez.com">
<blockquote type="cite"
cite="mid:7adfb521-88e2-75bc-8240-44d61c78b6ff@mtcc.com">
<blockquote type="cite"
cite="mid:CALOB-K_e0JxWhsk2RC=AvFdNnzFbbv66xfK0d0bVz7Qw=xuPFA@mail.gmail.com">
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<table
style="border-collapse:collapse;border-spacing:0px;margin:0px
0px
1.5rem;padding:0px;width:561px;font-size:12px;color:rgb(51,51,51);font-family:ff-clan-web-pro,"Helvetica
Neue",Helvetica,sans-serif">
<tbody>
<tr
style="border-bottom-style:none">
<td
style="padding:0px;font-weight:600;height:64px;width:64px;vertical-align:bottom"><br>
</td>
<td style="padding:20px
0px;width:24px"><br>
</td>
<td style="padding:20px
0px;width:24px"><br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</blockquote>
</blockquote>
</body>
</html>