<div dir="ltr">A nearby datacenter once lost power delayed because someone hit the switch to transfer from city power to generator power and then failed to notice. The power went out the day after when there was no fuel left.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 10, 2021 at 9:24 PM Matthew Huff <<a href="mailto:mhuff@ox.com">mhuff@ox.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_8842717686355071555WordSection1">
<p class="MsoNormal"><span>Since we are telling power horror stories…<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>How about the call from the night operator that arrived at 10:00pm asking “Is there any reason there is no power in the data
center?”<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Turns out someone had plugged in a new high end workgroup laser printer to the outside wall of the datacenter. The power receptacle
was wired into the data center’s UPS and completely smoked the UPS. Luckily the static transfer switched worked, but the three mainframes weren’t’ happy…<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Or<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>Our building had a major ground fault issue that took years to find and resolve. We got hit with lightning that caused the
mainframe to fault and recycle…and two minutes in, we got hit by lightning again. When the system failed to start, we called IBM support. When we explained what happened there was a very long pause…then some mumbling off phone, then the manager got on the
line and said someone would be flying out and be onsite within 12 hours. We were down for 3 days, and got fined $250,000 by the insurance regulators since we couldn’t pay claims.
<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-GB" style="font-family:Arial,sans-serif;color:rgb(31,73,125)">Matthew Huff</span></b><span lang="EN-GB" style="font-family:Arial,sans-serif;color:rgb(31,73,125)">
| Director of Technical Operations | OTA Management LLC<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-AU" style="font-size:4pt;font-family:Arial,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Arial,sans-serif;color:rgb(31,73,125)">Office: 914-460-4039<u></u><u></u></span></i></p>
<p class="MsoNormal"><i><span lang="EN-GB" style="font-family:Arial,sans-serif;color:rgb(31,73,125)"><a href="mailto:mhuff@ox.com" target="_blank"><span style="color:rgb(5,99,193)">mhuff@ox.com</span></a>
| </span></i><i><span lang="EN-GB" style="font-family:Arial,sans-serif"><a href="http://www.ox.com" target="_blank"><span style="color:rgb(5,99,193)">www.ox.com</span></a><span style="color:rgb(31,73,125)"><u></u><u></u></span></span></i></p>
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:Arial,sans-serif;color:gray">...........................................................................................................................................<u></u><u></u></span></b></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><b><span>From:</span></b><span> Chris Kane <<a href="mailto:ccie14430@gmail.com" target="_blank">ccie14430@gmail.com</a>>
<br>
<b>Sent:</b> Friday, September 10, 2021 3:16 PM<br>
<b>To:</b> Christopher Morrow <<a href="mailto:morrowc.lists@gmail.com" target="_blank">morrowc.lists@gmail.com</a>><br>
<b>Cc:</b> Matthew Huff <<a href="mailto:mhuff@ox.com" target="_blank">mhuff@ox.com</a>>; <a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a><br>
<b>Subject:</b> Re: Never push the Big Red Button (New York City subway failure)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in">True EPO story; maintenance crew carrying new drywall into the data center backed into the EPO that didn't have a cover on it. One of the most eerie sounds in networking...a completely silent data center.<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">-chris<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow <<a href="mailto:morrowc.lists@gmail.com" target="_blank">morrowc.lists@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <<a href="mailto:mhuff@ox.com" target="_blank">mhuff@ox.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12pt;margin-left:0.5in">
Reminds me of something that happened about 25 years ago when an elementary school visited our data center of the insurance company where I worked. One of our operators strategically positioned himself between the kids and the mainframe, leaned back and hit
it's EPO button.<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Or when your building engineering team cuts themselves a new key for the 'main breaker' for the facility... and tests it at 2pm on a tuesday.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Or when that same team cuts a second key (gotta have 2 keys!) and tests that key on the same 'main breaker' ... at 2pm on the following tuesday.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><quadruple face palm><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">not fakenews, a real story from a large building full of gov't employees and computers and all manner of 'critical infrastructure' for the agency occupying said building. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-left:0.5in">Matthew Huff | Director of Technical Operations | OTA Management LLC<br>
<br>
Office: 914-460-4039<br>
<a href="mailto:mhuff@ox.com" target="_blank">mhuff@ox.com</a> | <a href="http://www.ox.com" target="_blank">www.ox.com</a><br>
...........................................................................................................................................<br>
<br>
-----Original Message-----<br>
From: NANOG <nanog-bounces+mhuff=<a href="mailto:ox.com@nanog.org" target="_blank">ox.com@nanog.org</a>> On Behalf Of Sean Donelan<br>
Sent: Friday, September 10, 2021 12:38 PM<br>
To: <a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a><br>
Subject: Never push the Big Red Button (New York City subway failure)<br>
<br>
NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER<br>
OUTAGE ISSUE ON AUGUST 29, 2021<br>
Key Findings<br>
September 8, 2021<br>
<br>
<br>
<a href="https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf" target="_blank">https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf</a><br>
<br>
Key Findings<br>
[...]<br>
<br>
3. Based on the electrical equipment log readings and the manufacturer’s <br>
official assessment, it was determined that the most likely cause of RCC <br>
shutdown was the “Emergency Power Off” button being manually activated.<br>
<br>
Secondary Findings<br>
<br>
1. The “Emergency Power Off” button did not have a protective cover at the <br>
time of the shutdown or the following WSP investigation.<br>
<br>
[...]<br>
Mitigation Steps<br>
<br>
1. Set up the electrical equipment Control and Communication systems <br>
properly to stay active so that personnel can monitor RCC electrical <br>
system operations.<br>
<br>
[...]<u></u><u></u></p>
</blockquote>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:0.5in"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:0.5in"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:0.5in">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:0.5in">Chris Kane<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>