Famous operational issues

Fri Feb 19 07:25:08 UTC 2021

Northridge quake.  I was #2 and on call at CRL.  That One Guy on dialup in Atlanta playing MUDs 23x7 pages that things are down.  I wander out to my computer to dial in and see what’s up, turned on TV walking past it, sat down and turned computer on, as it was booting on comes a live helicopter shot over Northridge showing the 1.5 remaining floors of the 3-story Cable and Wireless building our east coast connector went through.

Took a second to listen and make sure I understood what was happening, changed channels to verify it wasn’t a stunt, logged  on and pinged our router there to confirm nothing there, call & wake up Jim: “East coast’s down because earthquake in Northridge and the C&W center fell down.”

“....oh.”

And then there was the Sidekick outage...

-George 

Sent from my iPhone

> On Feb 18, 2021, at 4:37 PM, Patrick W. Gilmore <patrick at ianai.net> wrote:
> 
> On Feb 18, 2021, at 6:10 PM, Karl Auer <kauer at biplane.com.au> wrote:
>> 
>> I think it was Macchiavelli who said that one should not ascribe to
>> malice anything adequately explained by incompetence…
> 
> https://en.wikipedia.org/wiki/Hanlon%27s_razor
>    Never attribute to malice that which is adequately explained by stupidity.
> 
> I personally prefer this version from Robert A. Heinlein:
>    Never underestimate the power of human stupidity.
> 
> And to put it on topic, cover your EPOs
> 
> In 1994, there was a major earthquake near the city of Los Angeles. City hall had to be evacuated and it would take over a year to reinforce the building to make it habitable again. My company moved all the systems in the basement of city hall to a new datacenter a mile or so away. After the install, we spent more than a week coaxing their ancient (even for 1994) machines back online, such as a Prime Computer and an AS400 with tons of DASD. Well, tons of cabinets, certainly less storage than my watch has now.
> 
> I was in the DC going over something with the lady in charge when someone walked in to ask her something. She said “just a second”. That person took one step to the side of the door and leaned against the wall - right on an EPO which had no cover.
> 
> Have you ever heard an entire row of DASD spin down instantly? Or taken 40 minutes to IPL an AS400? In the middle of the business day? For the second most populous city in the country?
> 
>    Me: Maybe you should get a cover for that?
>    Her: Good idea.
> 
> Couple weeks later, in the same DC, going over final checklist. A fedex guy walks in. (To this day, no idea how he got in a supposedly locked DC.) She says “just a second”, and I get a very strong deja vu feeling. He takes one step to the side and leans against the wall.
> 
>    Me: Did you order that EPO cover?
>    Her: Nope.
> 
> -- 
> TTFN,
> patrick
> 