Famous operational issues

Sat Feb 20 12:41:45 UTC 2021

Not a famous operational issue, but in 2000, we had a major outage of 
our dialup modem pool.

The owner of the building was re-skinning the outside using Styrofoam 
and stucco.  A bunch of the Styrofoam
had blocked the roof drains on the podium section of the building, 
immediately above our equipment room.

A flash rainstorm filled the entire flat roof, and water came back in 
over the flashings, and poured directly in
to our dialup modem pool through the hole in the concrete roof deck 
where the drain pipe protruded through.

In retrospect, it was a monumentally stupid place to put our main 
modem pool, but we didn't realize what was
above the drop ceiling - and that it was roof, not the other 11 
floors of the building.

1 bay of 6 shelves of USR TC 1000 HiperDSPs were now very wet and 
blinking funny patterns on their LEDs.

Fortunately, our vendor in Toronto (4 hour drive away) had stock of 
equipment that another customer kept
delaying shipment on.  They got their staff in, started un-boxing 
and, slotting cards.  We spent a few hours
tearing out the old gear and getting ready for replacements.

We left Windsor, Ontario at around 12:00am - same time they left 
Toronto, heading towards us.  We coordinated
a meet at one of the rural exits along Highway 401 at a closed gas 
station at around 2am.

Everything was going so well until a cop pulled up, and asked us what 
we were doing, as we were slinging
modem chassis between the back of the vendor's SUV and our van... We 
calmly explained
what happened.  He looked between us a couple of times, shook his 
head and said "well, good luck with that",
got back in his car and drove away.

We had everything back online within 14 hours of the initial outage.

At 02:37 PM 16/02/2021, John Kristoff wrote:
>Friends,
>
>I'd like to start a thread about the most famous and widespread Internet
>operational issues, outages or implementation incompatibilities you
>have seen.
>
>Which examples would make up your top three?
>
>To get things started, I'd suggest the AS 7007 event is perhaps  the
>most notorious and likely to top many lists including mine.  So if
>that is one for you I'm asking for just two more.
>
>I'm particularly interested in this as the first step in developing a
>future NANOG session.  I'd be particularly interested in any issues
>that also identify key individuals that might still be around and
>interested in participating in a retrospective.  I already have someone
>that is willing to talk about AS 7007, which shouldn't be hard to guess
>who.
>
>Thanks in advance for your suggestions,
>
>John

-- 

Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4

tel. 519-985-8410
fax. 519-985-8409        