massive facebook outage presently

Tue Oct 5 12:34:41 UTC 2021

I don't understand how this would have helped yesterday.

>From what is public so far, they really paint themselves in a corner with no way out. A classic, but at epic scale.

They will learn and improve for sure, but I don't understand how "firmware default to your own network" would have help here.

Can you elaborate a bit please?

Jean

-----Original Message-----
From: NANOG <nanog-bounces+jean=ddostest.me at nanog.org> On Behalf Of Glenn Kelley
Sent: October 4, 2021 8:18 PM
To: nanog at nanog.org
Subject: Re: massive facebook outage presently

This is why you should have Routers that are Firmware Defaulted to your own network.  ALWAYS

Be it Calix or even a Mikrotik which you have setup with Netboot - having these default to your own setup is REALLY a game changer.

Without it - you are rolling trucks or at minimum taking heavy call volumes.

Glenn Kelley
Chief cook and Bottle Washing Watcher @ Connectivity.Engineer

On 10/4/2021 3:56 PM, Matt Hoppes wrote:
> Yes,
> We've seen that.
>
> On 10/4/21 4:33 PM, Eric Kuhnke wrote:
>> I am starting to see reports that in ISPs with very large numbers of 
>> residential users, customers are starting to press the factory-reset 
>> buttons on their home routers/modems/whatever, in an attempt to make 
>> Facebook work. This is resulting in much heavier than normal first 
>> tier support volumes. The longer it stays down the worse this is 
>> going to get.
>>
>>
>>
>> On Mon, Oct 4, 2021 at 3:30 PM Jay Hennigan <jay at west.net 
>> <mailto:jay at west.net>> wrote:
>>
>>     On 10/4/21 12:11, bzs at theworld.com <mailto:bzs at theworld.com> wrote:
>>      >
>>      > Although I believe it's generally true that if a company appears
>>      > prominently in the news it's liable to be attacked I assume 
>> because
>>      > the miscreants sit around thinking "hmm, who shall we attack 
>> today oh
>>      > look at that shiny headline!" I'd hate to ascribe any altruistic
>>      > motivation w/o some evidence like even a credible twitter post 
>> (maybe
>>      > they posted that on FB? :-)
>>
>>     I personally believe that the outage was caused by human error 
>> and not
>>     something malicious. Time will tell.
>>
>>     However, if you missed the 60 Minutes piece, it was a former 
>> employee
>>     who spoke out with some rather powerful observations. I don't think
>>     that
>>     this type of worldwide outage was caused by an outside bad actor. 
>> It is
>>     certainly within the realm of possibility that it was an inside job.
>>
>>     In other news:
>>
>> https://twitter.com/disclosetv/status/1445100931947892736?s=20
>>
>>     --     Jay Hennigan - jay at west.net <mailto:jay at west.net>
>>     Network Engineering - CCIE #7880
>>     503 897-8550 - WB6RDV
>>