Never push the Big Red Button (New York City subway failure)

Fri Sep 10 18:48:05 UTC 2021

On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <mhuff at ox.com> wrote:

> Reminds me of something that happened about 25 years ago when an
> elementary school visited our data center of the insurance company where I
> worked. One of our operators strategically positioned himself between the
> kids and the mainframe, leaned back and hit it's EPO button.
>
>
Or when your building engineering team cuts themselves a new key for the
'main breaker' for the facility... and tests it at 2pm on a tuesday.
Or when that same team cuts a second key (gotta have 2 keys!) and tests
that key on the same 'main breaker' ... at 2pm on the following tuesday.

<quadruple face palm>

not fakenews, a real story from a large building full of gov't employees
and computers and all manner of 'critical infrastructure' for the agency
occupying said building.

Matthew Huff | Director of Technical Operations | OTA Management LLC
>
> Office: 914-460-4039
> mhuff at ox.com | www.ox.com
>
> ...........................................................................................................................................
>
> -----Original Message-----
> From: NANOG <nanog-bounces+mhuff=ox.com at nanog.org> On Behalf Of Sean
> Donelan
> Sent: Friday, September 10, 2021 12:38 PM
> To: nanog at nanog.org
> Subject: Never push the Big Red Button (New York City subway failure)
>
> NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
> OUTAGE ISSUE ON AUGUST 29, 2021
> Key Findings
> September 8, 2021
>
>
>
> https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf
>
> Key Findings
> [...]
>
> 3. Based on the electrical equipment log readings and the manufacturer’s
> official assessment, it was determined that the most likely cause of RCC
> shutdown was the “Emergency Power Off” button being manually activated.
>
> Secondary Findings
>
> 1. The “Emergency Power Off” button did not have a protective cover at the
> time of the shutdown or the following WSP investigation.
>
> [...]
> Mitigation Steps
>
> 1. Set up the electrical equipment Control and Communication systems
> properly to stay active so that personnel can monitor RCC electrical
> system operations.
>
> [...]
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210910/58acb36f/attachment.html>