Never push the Big Red Button (New York City subway failure)

Wed Sep 15 17:31:37 UTC 2021

On Wed, Sep 15, 2021 at 12:23 PM Daniel Seagraves <
dseagrav at humancapitaldev.com> wrote:

>
> On Sep 15, 2021, at 10:58 AM, Adam Thompson <athompson at merlin.mb.ca>
> wrote:
>
> Now I'm curious... in all of the DCs and COs I've worked in - to the best
> of my knowledge, I haven't personally tested this! - the EPO button does
> *not* switch to emergency power.  It turns off ALL equipment power in the
> space - no lights, no klaxons, nothing.  In simpler setups, the EPO is
> connected to the UPS so anything plugged in to the UPS does dark
> instantly.  In one DC I'm familiar with, the EPO switch kills all the UPS
> output *and* uses several relays to kill commercial power at the same
> time.
> In some, the room lights were not covered by the EPO switch, in some they
> were.  Emergency exit lamps will continue to be lit, as they have internal
> batteries, and are required by building/fire code.
>
>
> It was always my understanding EPO was to be used for “We have an
> electrical fire and need to remove the source RFN”, not “we need to be on
> the redundant power instead of city power and don’t want to wait for the
> automatic transfer”.
>
>
Well, there is the EPO button, which generally does that, and the
(variously labeled) HALON/FM-200/GAS FIRE SUPPRESSION/GAS DISCHARGE button,
which does the flashy lights and klangly bell and similar. This is fairly
much always required by code, to give people time to evacuate before the
gas dumps and they suffocate. People often refer to both of these as EPOs
(or "the buttons that must not be pressed unless you have a REALLY good
reason.").

When I grew up (in South Africa), Halon/BCF was still in active use. When
there was a fire (or you pressed and held the big red HALON button) a siren
would sound and lights would flash for a few seconds to allow everyone time
to evacuate the machine room.
I'm assuming that things are now less stupid, but at the local University,
the BCF was stored in large gas bottles, with a pyrotechnic valve to
release it. The pyrotechnic charge was initiated with LA/LS (Lead
Azide/Lead Styphnate) hot-wire initiators, which were supposed to be
replaced every 2 years as part of some maintenance schedule - when LA/LS
ages, especially in the presence of humidity, it apparently can form a much
more sensitive crystal structure, which is very shock sensitive.

The system was installed in the 1960s, and the initiators were replaced
once or twice. Eventually, however, with sanctions, especially on things
that can be made to go boom, it became hard to get replacements, and so
they stopped replacing them... and eventually forgot about them  ......
right up until sometime in the early 1990s, when someone accidentally
knocked into the bottles with a loaded equipment cart.
By this time the initiators had become sufficiently old and ornery that
they decided that they'd had enough, and set off the pyro charges, which
dumped all of the Halon into the room.

Luckily everyone survived, but IIRC, two people passed out before making it
to the door, and someone had to rush in and pull them to fresh air. The
added gas pressure also cracked the big glass window (what's the point in
having a big mainframe with flashy lights and spinning tapes if you cannot
show it off?), and also caused a few head-crashes.

W

-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210915/aae3d0ba/attachment.html>