Disaster Recovery Process

Tue Oct 5 17:25:41 UTC 2021

On Tue, Oct 5, 2021 at 1:07 PM Jeff Shultz <jeffshultz at sctcweb.com> wrote:

> 7. Make sure any access controlled rooms have physical keys that are
> available at need - and aren't secured by the same access control that they
> are to circumvent. .
> 8. Don't make your access control dependent on internet access - always
> have something on the local network  it can fall back to.
>
> That last thing, that apparently their access control failed, locking
> people out when either their outward facing DNS and/or BGP routes went
> goodbye, is perhaps the most astounding thing to me - making your access
> control into an IoT device without (apparently) a quick workaround for a
> failure in the "I" part.
>

Keep in mind that the "some employees couldn't get into their offices" has
been filtered through the public press and seems to have grown into "OMG!
Lolz! No-one can fix the Facebook because no-one can reach the
turn-it-off-and-on-again-button".
Facebook has many office buildings, and needs to be able to add and revoke
employee access as people are hired and quit, etc. Just because the press
said that some random employees were unable to enter their office building
doesn't actually mean that: 1: this was a datacenter and they really needed
access or 2: no-one was able to enter or 3: this actually caused issues
with recovery.
Important buildings have security people who have controller-locked cards
and / or physical keys, offices != datacenter, etc.

I'm quite sure that this part of the story is a combination of some small
tidbit of information that a non-technical reporter was able to understand,
mixed with some "Hah. Look at those idiots, even I know to keep a spare key
under the doormat" schadenfreude.

W

>
> On Tue, Oct 5, 2021 at 6:01 AM Jared Mauch <jared at puck.nether.net> wrote:
>
>>
>>
>> > On Oct 4, 2021, at 4:53 PM, Jorge Amodio <jmamodio at gmail.com> wrote:
>> >
>> > How come such a large operation does not have an out of bound access in
>> case of emergencies ???
>> >
>> >
>>
>> I mentioned to someone yesterday that most OOB systems _are_ the
>> internet.  It doesn’t always seem like you need things like modems or
>> dial-backup, or access to these services, except when you do it’s
>> critical/essential.
>>
>> A few reminders for people:
>>
>> 1) Program your co-workers into your cell phone
>> 2) Print out an emergency contact sheet
>> 3) Have a backup conference bridge/system that you test
>>   - if zoom/webex/ms are down, where do you go?  Slack?  Google meet?
>> Audio bridge?
>>   - No judgement, but do test the system!
>> 4) Know how to access the office and who is closest.
>>   - What happens if they are in the hospital, sick or on vacation?
>> 5) Complacency is dangerous
>>   - When the tools “just work” you never imagine the tools won’t work.
>> I’m sure the lessons learned will be long internally.
>>   - I hope they share them externally so others can learn.
>> 6) No really, test the backup process.
>>
>>
>>
>> * interlude *
>>
>> Back at my time at 2914 - one reason we all had T1’s at home was largely
>> so we could get in to the network should something bad happen.  My home IP
>> space was in the router ACLs.  Much changed since those early days as this
>> network became more reliable.  We’ve seen large outages in the past 2 years
>> of platforms, carriers, etc.. (the Aug 30th 2020 issue is still firmly in
>> my memory).
>>
>> Plan for the outages and make sure you understand your playbook.  It may
>> be from snow day to all hands on deck.  Test it at least once, and ideally
>> with someone who will challenge a few assumptions (eg: that the cell
>> network will be up)
>>
>> - Jared
>
>
>
> --
> Jeff Shultz
>
>
> Like us on Social Media for News, Promotions, and other information!!
>
>    <https://www.facebook.com/SCTCWEB/>     [image:
> https://www.instagram.com/sctc_sctc/]
> <https://www.instagram.com/sctc_sctc/>
> <https://www.yelp.com/biz/sctc-stayton-3>
> <https://www.youtube.com/c/sctcvideos>
>
>
>
>
>
>
>
> **** This message contains confidential information and is intended only
> for the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. ****
>

-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20211005/1294b893/attachment.html>