facebook outage

Patrick W. Gilmore patrick at ianai.net
Mon Oct 4 21:52:37 UTC 2021


On Oct 4, 2021, at 5:30 PM, Bill Woodcock <woody at pch.net> wrote:
> On Oct 4, 2021, at 11:21 PM, Bill Woodcock <woody at pch.net> wrote:
>> On Oct 4, 2021, at 11:10 PM, Bill Woodcock <woody at pch.net> wrote:
>>> 
>>> They’re starting to pick themselves back up off the floor in the last two or three minutes.  A few answers getting out.  I imagine it’ll take a while before things stabilize, though.
>> 
>> aaaand we’re back:
>> 
>> WoodyNet-2:.ssh woody$ dig www.facebook.com @9.9.9.9
> 
> So that was, what…  15:50 UTC to 21:05 UTC, more or less…  five hours and fifteen minutes.
> 
> That’s a lot of hair burnt all the way to the scalp, and some third-degree burns beyond that.
> 
> Maybe they’ll get one or two independent secondary authoritatives, so this doesn’t happen again.  :-)

If by “independent” you mean “3rd party” (e.g. DynDNS), not sure what an external secondary would have done here. While their BGP was misbehaving, the app would not work even if you had a static DNS entry.

And while using external / 3rd party secondaries is likely a good idea for many companies, almost none of the largest do this. These companies view it as a control issue. Giving someone outside your own employees the ability to change a DNS name is, frankly, giving another company the ability to take you down.

Taking a sample of FB, cisco, Amazon, NF, Dell, Akamai, Google, MS, CF, only 2 use 3rd party resolvers.
* NF uses only awsdns, so same problem, just moved to another company they do not control.
* Amazon uses Ultra & Dyn. (Anyone else amused amazon.com has no authorities on Route 53? At least not from my vantage point.)

That said, plenty of what people may call “big” companies do use 3rd parties, e.g. IBM, PayPal, Juniper.

You want to use a 3rd party DNS, go for it. There are lots of reasons to do it. But it is not a panacea, and there are reasons not to.

-- 
TTFN,
patrick



More information about the NANOG mailing list