Famous operational issues

Patrick W. Gilmore patrick at ianai.net
Fri Feb 19 00:34:39 UTC 2021


On Feb 18, 2021, at 6:10 PM, Karl Auer <kauer at biplane.com.au> wrote:
> 
> I think it was Macchiavelli who said that one should not ascribe to
> malice anything adequately explained by incompetence…

https://en.wikipedia.org/wiki/Hanlon%27s_razor
	Never attribute to malice that which is adequately explained by stupidity.

I personally prefer this version from Robert A. Heinlein:
	Never underestimate the power of human stupidity.

And to put it on topic, cover your EPOs

In 1994, there was a major earthquake near the city of Los Angeles. City hall had to be evacuated and it would take over a year to reinforce the building to make it habitable again. My company moved all the systems in the basement of city hall to a new datacenter a mile or so away. After the install, we spent more than a week coaxing their ancient (even for 1994) machines back online, such as a Prime Computer and an AS400 with tons of DASD. Well, tons of cabinets, certainly less storage than my watch has now.

I was in the DC going over something with the lady in charge when someone walked in to ask her something. She said “just a second”. That person took one step to the side of the door and leaned against the wall - right on an EPO which had no cover.

Have you ever heard an entire row of DASD spin down instantly? Or taken 40 minutes to IPL an AS400? In the middle of the business day? For the second most populous city in the country?

	Me: Maybe you should get a cover for that?
	Her: Good idea.

Couple weeks later, in the same DC, going over final checklist. A fedex guy walks in. (To this day, no idea how he got in a supposedly locked DC.) She says “just a second”, and I get a very strong deja vu feeling. He takes one step to the side and leans against the wall.

	Me: Did you order that EPO cover?
	Her: Nope.

-- 
TTFN,
patrick



More information about the NANOG mailing list