Never push the Big Red Button (New York City subway failure)

Daniel Seagraves dseagrav at humancapitaldev.com
Wed Sep 15 22:45:03 UTC 2021


> On Sep 15, 2021, at 2:20 PM, Fred Baker <fredbaker.ietf at gmail.com> wrote:
> 
> One of the many stories that came out of 9/11 was a switching center in NY City that had a diesel generator as a power backup - which of course acted as primary when the city power is off. After a few days of operation, it needed to be refueled, so a truck was sent in carrying gasoline. The generator was refueled and restarted, and - oops - diesel != gasoline. So then they needed to bring in a new generator.
> 
> Yup, it happens, and it happened.

I distinctly remember something like this - Someone built a datacenter with large fuel storage tanks in the basement and the actual generators up on the roof, or some higher floor. It was tested several times, everything seemed to be working as expected, and life went on. Then one day the power went out, the generators came on, but after about 10 minutes the generators started to crap out. It was then discovered that they had forgotten to include the transfer pumps for getting the fuel up from the basement to the generators in the list of things powered by said generators…



More information about the NANOG mailing list