DHS letters for fuel and facility access

Tue Mar 17 17:33:04 UTC 2020

On Tue, Mar 17, 2020 at 1:21 PM Hiers, David <David.Hiers at cdk.com> wrote:
>
> Good reminder to test, test, test...

Indeed -- and we had tested, multiple times. Unfortunately, the only
realistic way we would have found this would have been to kill power
to the building and run on generators for many hours, and then,
likely, we would only have discovered it when the gensets ran out of
power and fell over. IIRC, there is (or was) some noise and pollution
regulations in NYC where you could only run generators for short
periods of time (30min?) unless it was an actual emergency. I also
seem to remember something about having to test at night, probably
also for noise...

But, yes, regular testing is clearly a good practice - but so is
having a good BCP/DR plan (which you also test :-)
W

>
>
> -----Original Message-----
> From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Warren Kumari
> Sent: Tuesday, March 17, 2020 10:08 AM
> To: Paul Nash <paul at nashnetworks.ca>
> Cc: Untitled 3 <nanog at nanog.org>
> Subject: Re: DHS letters for fuel and facility access
>
> On Tue, Mar 17, 2020 at 12:44 PM Paul Nash <paul at nashnetworks.ca> wrote:
> >
> > September 2001.  Just after the 9/11 attacks, all of lower Manhattan was shut down.  Out link (IIRC) was to a satellite farm on Staten island, across the bay to 60 Hudson.  Power went off, diesels kicked in, fuel trucks was not allowed in, and a few days later we lost all international connectivity.
>
> We had some interesting failures during 9/11 as well -- for some reason, the UPS didn't kick in, so everything went down - and then came back a few minutes later as the generators came online -- and then went down again ~2 hours later -- turns out that the genset air filters got clogged with dust, and suffocated the diesel.
> This was "fixed" a few days later by brushing them off with brooms and paintbrushes -- by this point they had completely discharged the 24V starter batteries, and so someone (not me!) had to lug a pair of car batteries and jumper cables. They restarted, and ran for a while, and then stopped again.
>
> It turns out that getting a permit to store lots of diesel on the roof is hard (fair enough), and so there was only a small holding tank on the roof, and the primary tanks were in the basement -- and the transfer pump from the basement to roof storage was not, as we had been told, on generator power....
>
> We had specified that the transfer pump be on the generator feed, there was a schematic showing at is being on the generator feed, there was even a breaker with a cable marked  "Transfer Pump (HP4,5)" --- but it turned out to just be a ~3ft piece of cable stuffed into a conduit, and not actually, you know, running all the way down to the basement and connected to the transfer pump.
>
> W
>
>
>
> >
> > Lots of important people lost power as well, so the feds decided to let the diesel tankers in after a few days’ deliberations.
> >
> >         paul
> >
> > > On Mar 17, 2020, at 11:21 AM, Mark Tinka <mark.tinka at seacom.mu> wrote:
> > >
> > >
> > >
> > > On 17/Mar/20 17:15, Paul Nash wrote:
> > >
> > >> That same fuel shortage killed all Internet traffic to sub-Saharan Africa.  Took us a while to figure out what was wrong with the satellite link to the US.
> > >
> > > What year was that :-)?
> > >
> > > Mark.
> >
>
>
> --
> I don't think the execution is relevant when it was obviously a bad idea in the first place.
> This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants.
>    ---maf
>
> ----------------------------------------------------------------------
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf