Don't press the big red buttom on the wall!
larrysheldon at cox.net
Wed Aug 31 00:52:40 UTC 2016
On 8/30/2016 09:40, Keith Stokes wrote:
> At one point in one data center I dealt with a disgruntled employee
> hit the UPS disconnect button on the way out.
> Same story, procedures modified, cover put over switch with a hammer
> to break the glass, lessons learned, accounts credited.
A very long time ago ("network" involved a fleet of green "wide-band"
trucks, hauling tapes to contractors and other offices) the system
involved 9 computer centers around the state, built over a period of
years, so they had a lot of similarities but some key differences.
Many of them had wide, pneumatic sliding doors between the computer room
and the unit-record rooms. Some of the doors had floor mats that would
pop the doors open when stepped on (or a cart full of card trays was
Many of them (for what ever reason--I think I know but it isn't relevant
here) had large black buttons on each side of the doors, on each side of
It happened that one had the mats at the sliding doors. but there was an
ordinary door near the consoles that had a large black button next to it.
It was in this office that a conversion team was running some stuff that
ran for hours (in violation of the rule that if a job ran more than
thirty minutes it Must Include checkpoint-restart points every 20
minutes) was nearly finished after running all day and all night (as I
One of the team left the computer room via the ordinary door, pushing
the big black button.
Which was (you saw this coming a long time ago, right?) the Emergency
Power Off button.
I do not recall any lessons being learned. At all. The group leader
(that refused to include checkpoint-restart) years later was conducting
a conversion run in a different system but that had many of the same
standards ran a job that ran many many hours in a computer center known
for flaky power. Without Checkpoint-Restart. We took a power hit when
the run had something like 24 records left to process.
"Everybody is a genius. But if you judge a fish by
its ability to climb a tree, it will live its whole
life believing that it is stupid."
From Larry's Cox account.
More information about the NANOG