Famous operational issues

Thu Feb 18 20:31:57 UTC 2021

A few I remember:

     . Some monitoring server SCSI drive failed (we're talking 
State/Province level govt)...  Got a return back stating it will take 6 
month delay to get a replacement...

         Ended up choosing to use my own drive instead of leaving 
something that could be have been deadly, unmonitored.

     . Metro interruption during rush hour (for a pop of 4M) due to 
overload power bar in a MMR (Meet Me Room) during a unplanned deployment;

     . Cherry red and very angry looking 520-600V bus bar =D;

     . Fire fighters hitting the building generator emergency STOP 
button because some neighbor reported smoke on top of the building 
during a black out...
     ( not their fault, local gov failure as usual )

     . Some idiots poured gasoline into a large pipe under a bridge...  
ended up demonstrating the lack of diversity to the DCs on that urban 
island;

     . Underground transformer blow up downtown Mtl and took out the 
entire fiber bundle, demonstrating to those customers that their 
diversity was actually real =D.

         (took them a year to get that fixed)

and

     . Obviously: Any rack cabling I do...

-----
Alain Hebert                                ahebert at pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770     Beaconsfield, Quebec     H9W 6G7
Tel: 514-990-5911  http://www.pubnix.net    Fax: 514-990-9443

On 2/18/21 2:37 PM, tim at pelican.org wrote:
> On Thursday, 18 February, 2021 16:23, "Seth Mattinen" <sethm at rollernet.us> said:
>
>> I had a customer that tried to stack their servers - no rails except the
>> bottom most one - using 2x4's between each server. Up until then I
>> hadn't imagined anyone would want to fill their cabinet with wood, so I
>> made a rule to ban wood and anything tangentially related (cardboard,
>> paper, plastic, etc.). Easier to just ban all things. Fire reasons too
>> but mainly I thought a cabinet full of wood was too stupid to allow.
> On the "stupid racking" front, I give you most of a rack dedicated to a single server.  Not all that high a server, maybe 2U or so, but *way* too deep for the rack, so it had been installed vertically.  By looping some fairly hefty chain through the handles on either side of the front of the chassis, and then bolting the four chain ends to the four rack posts.  I wish I'd kept pictures of that one.  Not flammable, but a serious WTF moment.
>
> Cheers,
> Tim.
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210218/8e248347/attachment.html>