Never push the Big Red Button (New York City subway failure)

Fri Sep 10 19:16:22 UTC 2021

True EPO story; maintenance crew carrying new drywall into the data center
backed into the EPO that didn't have a cover on it. One of the most
eerie sounds in networking...a completely silent data center.

-chris

On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow <morrowc.lists at gmail.com>
wrote:

>
>
> On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <mhuff at ox.com> wrote:
>
>> Reminds me of something that happened about 25 years ago when an
>> elementary school visited our data center of the insurance company where I
>> worked. One of our operators strategically positioned himself between the
>> kids and the mainframe, leaned back and hit it's EPO button.
>>
>>
> Or when your building engineering team cuts themselves a new key for the
> 'main breaker' for the facility... and tests it at 2pm on a tuesday.
> Or when that same team cuts a second key (gotta have 2 keys!) and tests
> that key on the same 'main breaker' ... at 2pm on the following tuesday.
>
> <quadruple face palm>
>
> not fakenews, a real story from a large building full of gov't employees
> and computers and all manner of 'critical infrastructure' for the agency
> occupying said building.
>
> Matthew Huff | Director of Technical Operations | OTA Management LLC
>>
>> Office: 914-460-4039
>> mhuff at ox.com | www.ox.com
>>
>> ...........................................................................................................................................
>>
>> -----Original Message-----
>> From: NANOG <nanog-bounces+mhuff=ox.com at nanog.org> On Behalf Of Sean
>> Donelan
>> Sent: Friday, September 10, 2021 12:38 PM
>> To: nanog at nanog.org
>> Subject: Never push the Big Red Button (New York City subway failure)
>>
>> NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
>> OUTAGE ISSUE ON AUGUST 29, 2021
>> Key Findings
>> September 8, 2021
>>
>>
>>
>> https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf
>>
>> Key Findings
>> [...]
>>
>> 3. Based on the electrical equipment log readings and the manufacturer’s
>> official assessment, it was determined that the most likely cause of RCC
>> shutdown was the “Emergency Power Off” button being manually activated.
>>
>> Secondary Findings
>>
>> 1. The “Emergency Power Off” button did not have a protective cover at
>> the
>> time of the shutdown or the following WSP investigation.
>>
>> [...]
>> Mitigation Steps
>>
>> 1. Set up the electrical equipment Control and Communication systems
>> properly to stay active so that personnel can monitor RCC electrical
>> system operations.
>>
>> [...]
>>
>

-- 
Chris Kane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210910/c0894a8f/attachment.html>