Never push the Big Red Button (New York City subway failure)

Thu Sep 16 12:19:40 UTC 2021

I once worked for a provider who had a company next door that ran a small datacenter of about a dozen or so racks.    They had been sold and all of their infrastructure had been virtualized and moved to the new owner’s network.    The last task of the local admin was to just get rid of everything.   They didn’t care how just get it gone.    So he came over and asked us to come take a look and we could have anything we wanted.    I picked up a few servers for lab a bunch of racks and stuff.

While we were working I asked the guy  “So there is absolutely nothing that’s in production in here anymore?”    He said “Yep” so I asked “Then if the power went off in here it wouldn’t be a big deal”   and he said “Not at all”.    Then I asked “Can I hit the red button?”    He said “Sure, I always wondered what happened”.     I hit the button and with a loud booming sound the room went dead silent and then the UPS started beeping.      It was at that moment everyone realized that you just don’t pull the button out to restart the room.      It took us 20 minutes to figure out how to turn it all back on.

And with that when I got back to our office I made sure someone knew how to restart everything if we ever had to hit our red button.

-richey

From: NANOG <nanog-bounces+richey.goldberg=gmail.com at nanog.org> on behalf of Roy <r.engehausen at gmail.com>
Date: Thursday, September 16, 2021 at 12:41 AM
To: nanog <nanog at nanog.org>
Subject: Re: Never push the Big Red Button (New York City subway failure)
Miy story in the late 1970s I was working in a large computer facility
with both mainframes and mil-spec 400hz computers.
Management decided that the EPO should be tested.  So we powered down
the disk and tapes.  The electrician pressed
the EPO button and NOTHING.  Everything kept running.

Turns out a wire had come loose and the fuse in the EPO circuit had blown.

Roy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210916/63c5da29/attachment.html>