Don't press the big red buttom on the wall!

Wed Aug 31 00:52:40 UTC 2016

On 8/30/2016 09:40, Keith Stokes wrote:
> At one point in one data center I dealt with a disgruntled employee
> hit the UPS disconnect button on the way out.
>
> Same story, procedures modified, cover put over switch with a hammer
> to break the glass, lessons learned, accounts credited.

A very long time ago ("network" involved a fleet of green "wide-band" 
trucks, hauling tapes to contractors and other offices) the system 
involved 9 computer centers around the state, built over a period  of 
years, so they had a lot of similarities but some key differences.

Many of them had wide, pneumatic sliding doors between the computer room 
and the unit-record rooms.  Some of the doors had floor mats that would 
pop the doors open when stepped on (or a cart full of card trays was 
rolled onto).

Many of them (for what ever reason--I think I know but it isn't relevant 
here) had large black buttons on each side of the doors, on each side of 
the wall.

It happened that one had the mats at the sliding doors. but there was an 
ordinary door near the consoles that had a large black button next to it.

It was in this office that a conversion team was running some stuff that 
ran for hours (in violation of the rule that if a job ran more than 
thirty minutes it Must Include checkpoint-restart points every 20 
minutes) was nearly finished after running all day and all night (as I 
recall it).

One of the team left the computer room via the ordinary door, pushing 
the big black button.

Which was (you saw this coming a long time ago, right?) the Emergency 
Power Off button.

I do not recall any lessons being learned.  At all.  The group leader 
(that refused to include checkpoint-restart) years later was conducting 
a conversion run in a different system but that had many of the same 
standards ran a job that ran many many hours in a computer center known 
for flaky power.  Without Checkpoint-Restart.  We took a power hit when 
the run had something like 24 records left to process.

-- 
"Everybody is a genius.  But if you judge a fish by
its ability to climb a tree, it will live its whole
life believing that it is stupid."

--Albert Einstein

 From Larry's Cox account.