Never push the Big Red Button (New York City subway failure)

Fred Baker fredbaker.ietf at gmail.com
Wed Sep 15 19:20:21 UTC 2021



> On Sep 10, 2021, at 1:33 PM, Warren Kumari <warren at kumari.net> wrote:
> 
> The utility let them know that they were going to be doing some maintenance work in the area. No impact expected, but out of an abundance of caution, they transfer over to generators. After the utility lets them know that the maintenance work is all finished, they want to switch back. If the generators are "emergency power", and you need to switch back to "utility power", obviously the way to do this must be the big red button, clearly marked as "EMERGENCY POWER OFF", no?!

One of the many stories that came out of 9/11 was a switching center in NY City that had a diesel generator as a power backup - which of course acted as primary when the city power is off. After a few days of operation, it needed to be refueled, so a truck was sent in carrying gasoline. The generator was refueled and restarted, and - oops - diesel != gasoline. So then they needed to bring in a new generator.

Yup, it happens, and it happened.


More information about the NANOG mailing list