Famous operational issues

Tue Feb 23 00:58:29 UTC 2021

Let me tell you about my personal favorite.

It’s 2002 and I am working as an engineer for an electronic stock trading platform (ECN), this platform happened to be the biggest platform for trading stocks electronically, on some days bigger than NASDAQ itself. This platform also happened to be run on DOS, FoxPro and a Novell file share, on a cluster of roughly 1,000 computers, two of which were the “engine” that matched all of the trades.

Well FoxPro has this “feature” where the ESC key halts the running program. We had the ability to remote control these DOS/FoxPro machines via some program we had written. Someone asked me to check the status of the process running on the primary matching engine, and when I was done, out of habit, I hit ESC. Trade processing grinds to a halt (phone calls have to be made to the SEC). I immediately called the NOC and told them it was me. Next thing I know, someone from the NOC is at my desk with a screwdriver putting the ESC key from my keyboard. I remained ESC keyless for the next several years until I left the company. I was hazed pretty good over it, but was essentially given a one time pass.

> On Feb 22, 2021, at 7:30 PM, bzs at theworld.com wrote:
> 
> 
> At Boston Univ we discovered the hard way that a security guard's
> walkie-talkie could cause a $5,000 (or $10K for the big machine room)
> Halon dump.
> 
> Took a couple of times before we figured out the connection tho once
> someone made it to the hold button before it actually dumped.
> 
> Speaking of halon one very hot day I'm goofing off drinking coffee at
> a nearby sub shop when the owner tells me someone from the computing
> center was on the phone, that never happened before.
> 
> Some poor operator was holding the halon shot, it's a deadman's switch
> (well, button) and the building was doing its 110db thing could I come
> help? The building is being evac'd.
> 
> So my boss who wasn't the sharpest knife in the drawer follows me down
> as I enter and I'm sweating like a pig with a floor panel sucker
> trying to figure out which zone tripped.
> 
> And he shouts at me over the alarms: WHY TF DOES IT DO THIS?! Angrily.
> 
> I answered: well, maybe THERE'S A FIRE!!!
> 
> At which point I notice the back of my shoulder is really bothering
> me, which I say to him, and he says hmmm there's a big bee on your
> back maybe he's stinging you?
> 
> Fun day.
> 
> -- 
>        -Barry Shein
> 
> Software Tool & Die    | bzs at TheWorld.com             | http://www.TheWorld.com
> Purveyors to the Trade | Voice: +1 617-STD-WRLD       | 800-THE-WRLD
> The World: Since 1989  | A Public Information Utility | *oo*