<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 9, 2021 at 1:40 AM Masataka Ohta <<a href="mailto:mohta@necom830.hpcl.titech.ac.jp">mohta@necom830.hpcl.titech.ac.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Christopher Morrow wrote:<br>
>> means their DNS servers were serving the zone, even after they<br>
>> recognize their zone data were too old, that is, expired.<br>
<br>
> that's not what this means. I think Mr. Petach previously described<br>
> this,<br>
<br>
He wrote:<br>
<br>
> So, the idea is that if the edge CDN node loses connectivity to<br>
> the core datacenters, the DNS servers should stop answering<br>
> queries for A records with the local CDN node's address, and<br>
> let a different site respond back to the client's DNS request.<br>
<br>
which may be performed by standard DNS with short expire period,<br>
after which name servers will return SERVFAIL and other name<br>
servers in other edge node with different IP addresses are tried.<br></blockquote><div><br></div><div>(Apologies for the delayed response--I had back-to-back board </div><div>meetings the past two days which had me completely tied up.)</div><div><br></div><div>That is one way in which it *could* be done--but is by no means </div><div>the ONLY way in which it can be done.</div><div><br></div><div>With an anycast setup using the same IP addresses in every </div><div>location, returning SERVFAIL doesn't have the same effect, </div><div>however, because failing over from anycast address 1 to </div><div>anycast address 2 is likely to be routed to the same pop </div><div>location, where the same result will occur.</div><div><br></div><div>You don't really want to hunt among different *IP addresses*,</div><div>you want to hunt to a different *location*.</div><div><br></div><div>This is why withdrawing the BGP announcement from that </div><div>location works more effectively, because it allows the clients </div><div>to continue querying the same IP address, but get routed to </div><div>the next most proximal location.</div><div><br></div><div>If you simply return SERVFAIL and have the client pick a </div><div>different IP address from the list of NS entries, it falls into </div><div>one of two situations:</div><div>a) the new IP address is also anycasted, and is therefore </div><div> likely to pick the same pop that is unhealthy, with similar</div><div> results, or</div><div>b) the new IP address is *not* anycasted, but is served from </div><div> a single geographical location, which means answers given </div><div> back by that DNS server are unlikely to be geolocated with </div><div> any accuracy, and therefore the content served is also unlikely </div><div> to be geographically relevant or correct.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
It may be that facebook uses all the four name server IP addresses<br>
in each edge node. But, it effectively kills essential redundancy<br>
of DNS to have two or more name servers (at separate locations)<br>
and the natural consequence is, as you can see, mass disaster.<br></blockquote><div><br></div><div>Even if the four anycasted nameserver IP addresses weren't </div><div>completely overlapping (let's assume as a hypothetical that </div><div>a.ns is served out of EU pops, b.ns is served out of NA pops,</div><div>c.ns is served out of SA pops, and d.ns is served out of APAC </div><div>pops), if all sites run the same healthcheck code, then if the </div><div>underlying healthcheck fails, *every site* will decide it is </div><div>unhealthy, and stop answering requests; so, all the EU sites </div><div>fail health check and stop serving a.ns; all the North America </div><div>sites fail health check, and stop serving b.ns...and so forth.</div><div><br></div><div>You followed the best practices, you had different NS entries </div><div>that were on different subnets, that were geographically </div><div>dispersed around the globe, that were redundant for each </div><div>other. But because they all used the same fundamental </div><div>health check, they all *independently* decided they were </div><div>unhealthy and needed to stop giving out DNS answers, </div><div>and instead let one of the other healthier sites take over.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> but: 1) dns server in pop serves some content (ttls aren't<br>
> important right now)<br>
<br>
You MUST distinguish TTL and EXPIRE. They are different.<br></blockquote><div><br></div><div>TTL and EXPIRE are irrelevant here.</div><div>The only thing changing those values would do is change </div><div>how long it took for caching resolvers to reflect the loss of </div><div>connectivity at the DNS layer. Once the underlying layer 3 </div><div>connectivity had broken, DNS answers became meaningless.</div><div>No matter what records were returned, or cached, you couldn't </div><div>reach the servers.</div><div><br></div><div>Yes, yes, as an academic exercise you can point out that </div><div>there's a difference in how and when those DNS records </div><div>stop being used, and you're right about that--but in terms </div><div>of this particular failure, this particular post-mortem we're </div><div>beating to a horse-shaped pulp, it's entirely meaningless. ^_^;</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> there's not a lot of magic here... and it's not about the zone data<br>
> really at all.<br>
<br>
Statement of Petach: "the edge CDN node loses connectivity to<br>
the core datacenters, the DNS servers should stop answering"<br>
means, with DNS terminology, zone data is expired, which has<br>
nothing to do with TTL.<br></blockquote><div><br></div><div>As you're using my words, I'm going to have to point out that</div><div>"the DNS servers should stop answering" does not require that </div><div>any change happens *at the DNS layer* -- in this case, the </div><div>change can happen at the routing layer, ensuring that even </div><div>if some caching resolver out there is completely defiant of </div><div>your expire time, you *will not answer* because the query </div><div>packets can never reach you in the first place.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Masataka Ohta<br></blockquote><div><br></div><div>Thanks!</div><div><br></div><div>Matt</div><div> </div></div></div>