massive facebook outage presently

Łukasz Bromirski lukasz at bromirski.net
Mon Oct 4 20:27:57 UTC 2021


Dual homing won’t help you if your automation template will do „no router bgp X” and at this point session will terminate as suddenly advertisement will be withdrawn…

It won’t you either if the change triggers some obscure bug in your BGP stack.

I bet FB tested the change on smaller scale and everything was fine, and only then started to roll this over wider network and at that point „something” broke. Or some bug needed a moment to start cascading issues around the infra.

-- 
./

> On 4 Oct 2021, at 22:00, Michael Thomas <mike at mtcc.com> wrote:
> 
> 
> 
> 
> On 10/4/21 11:48 AM, Luke Guillory wrote:
>> 
>> I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally.
> Assuming that this is what actually happened, what should fb have done different (beyond the obvious of not screwing up the immediate issue)? This seems like it's a single point of failure. Should all of the BGP speakers have been dual homed or something like that? Or should they not have been mixing ops and production networks? Sorry if this sounds dumb.
> 
> Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20211004/d1762dba/attachment.html>


More information about the NANOG mailing list