San Francisco Power Outage

Wed Jul 25 03:58:16 UTC 2007

On Tue, 24 Jul 2007, Tuc at T-B-O-H.NET wrote:

> (I remember two guys with VERY LONG screwdrivers poking a live transfer
> switch to get it to reset properly, and was told to step back 20 feet as
> thats how far they expected to get thrown if they did something wrong).
> (I also remember them resetting the switch, then TRIPPING it again just
> to make sure it could be reset again!)

Ahhh, a trip down memory lane :)

The ISP I used to work at had a small ping-and-power colo space, and we 
also housed a large dial/DSL POP in the same building.  A customer went in 
to do hardware maintenance on one of their colo boxes.  Two important 
notes here:

1. The machine was still plugged in to the power outlet when they decided 
to do this work.
2. They decided to stick a screwdriver into the power supply WHILE said 
machine was plugged into said power outlet.  I guess those "no user 
serviceable parts inside" warning labels are just friendly recommendations 
and nothing more...

While the machine was fed from a circuit that other colo customers were 
on, the breaker apparently didn't trip quickly enough to keep the 
resulting short from sending the 20 kva Liebert UPS at the back of the 
room into a fit.  It alarmed then shut down within 1-2 seconds of this 
customer doing the trick with the screwdriver.  This UPS also fed said 
large dial and DSL POP.  Nothing quite like the sound of a whole machine 
room spinning down at the same time.  It gives you that lovely "oh shit" 
feeling in the pit of your stomach.

I do remember fighting back the urge to stab said customer with that 
screwdriver...

jms