Disaster Recovery Process
jim deleskie
deleskie at gmail.com
Tue Oct 5 17:09:18 UTC 2021
World broke. Crazy $$ per hour down time. Doors open with a fire axe.
Glass breaks super easy too and much less expensive then adding 15 min to
failure.
-jim
On Tue., Oct. 5, 2021, 7:05 p.m. Jeff Shultz, <jeffshultz at sctcweb.com>
wrote:
> 7. Make sure any access controlled rooms have physical keys that are
> available at need - and aren't secured by the same access control that they
> are to circumvent. .
> 8. Don't make your access control dependent on internet access - always
> have something on the local network it can fall back to.
>
> That last thing, that apparently their access control failed, locking
> people out when either their outward facing DNS and/or BGP routes went
> goodbye, is perhaps the most astounding thing to me - making your access
> control into an IoT device without (apparently) a quick workaround for a
> failure in the "I" part.
>
> On Tue, Oct 5, 2021 at 6:01 AM Jared Mauch <jared at puck.nether.net> wrote:
>
>>
>>
>> > On Oct 4, 2021, at 4:53 PM, Jorge Amodio <jmamodio at gmail.com> wrote:
>> >
>> > How come such a large operation does not have an out of bound access in
>> case of emergencies ???
>> >
>> >
>>
>> I mentioned to someone yesterday that most OOB systems _are_ the
>> internet. It doesn’t always seem like you need things like modems or
>> dial-backup, or access to these services, except when you do it’s
>> critical/essential.
>>
>> A few reminders for people:
>>
>> 1) Program your co-workers into your cell phone
>> 2) Print out an emergency contact sheet
>> 3) Have a backup conference bridge/system that you test
>> - if zoom/webex/ms are down, where do you go? Slack? Google meet?
>> Audio bridge?
>> - No judgement, but do test the system!
>> 4) Know how to access the office and who is closest.
>> - What happens if they are in the hospital, sick or on vacation?
>> 5) Complacency is dangerous
>> - When the tools “just work” you never imagine the tools won’t work.
>> I’m sure the lessons learned will be long internally.
>> - I hope they share them externally so others can learn.
>> 6) No really, test the backup process.
>>
>>
>>
>> * interlude *
>>
>> Back at my time at 2914 - one reason we all had T1’s at home was largely
>> so we could get in to the network should something bad happen. My home IP
>> space was in the router ACLs. Much changed since those early days as this
>> network became more reliable. We’ve seen large outages in the past 2 years
>> of platforms, carriers, etc.. (the Aug 30th 2020 issue is still firmly in
>> my memory).
>>
>> Plan for the outages and make sure you understand your playbook. It may
>> be from snow day to all hands on deck. Test it at least once, and ideally
>> with someone who will challenge a few assumptions (eg: that the cell
>> network will be up)
>>
>> - Jared
>
>
>
> --
> Jeff Shultz
>
>
> Like us on Social Media for News, Promotions, and other information!!
>
> <https://www.facebook.com/SCTCWEB/> [image:
> https://www.instagram.com/sctc_sctc/]
> <https://www.instagram.com/sctc_sctc/>
> <https://www.yelp.com/biz/sctc-stayton-3>
> <https://www.youtube.com/c/sctcvideos>
>
>
>
>
>
>
>
> **** This message contains confidential information and is intended only
> for the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. ****
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20211005/3843d22d/attachment.html>
More information about the NANOG
mailing list