UUNET connectivity in Minneapolis, MN

Warren Kumari Warren at kumari.net
Fri Aug 12 16:58:41 UTC 2005


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So I am standing in a datacenter fiddling with some fiber and  
listening to an electrician explaining to the datacenter owner how he  
has just finished auditing all of the backup power systems and that  
the transfer switch will work this time (unlike the last 3 times).  
This is making me a little nervous, but I keep quiet (unusual for  
me)... Electrician starts walking out of the DC, looks at the  
(glowing) Big Red Button (marked "Emergency Power Off") and says  
"Hey, why ya'll running on emergency power?" and presses BRB. Lights  
go dark, disks spin down, Warren takes his business elsewhere!

This is the same DC that had large basement mounted generators in a  
windowless building in NYC.  Weeks before the above incident they had  
tried to test the generator (one of the failed transfer switch  
incidents), but apparently no one knew that there were manual flues  
at the top of the exhausts.... Carbon monoxide, building evacuated...

Warren

On Aug 12, 2005, at 8:27 AM, Valdis.Kletnieks at vt.edu wrote:

> On Fri, 12 Aug 2005 06:50:47 CDT, "James D. Butt" said:
>
>
>> Unless there is some sort of crazy story related to why a service  
>> provider
>> could not keep the lights on, this should have not been an issue with
>> proper operations and engineering.
>>
>
> So a while ago, we're in the middle of some major construction to  
> put in
> infrastructure for a supercomputer.  Meanwhile, as an unrelated  
> project we
> installed a new diesel backup generator to replace an older  
> generator that was
> undersized for our current systems, and take several hours of downtime
> on a Saturday to wire the beast in.
>
> The next Friday, some contractors are moving the entrance to our  
> machine room
> about 30 feet to the right, so you don't walk into the middle of the
> supercomputer.  Worker A starts moving a small red switch unit from  
> its
> location next to where the door used to be to its new location next  
> to where
> the door was going to be.  Unfortunately, he did it before double- 
> checking with
> Worker B that the small red switch was disarmed...
>
> Ka-blammo, a Halon dump... and of course that's interlocked with  
> the power,
> so once the Halon stopped hissing, it was *very* quiet in there.....
>
> Moral: It only takes one guy with a screwdriver.....
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iD8DBQFC/NVFHSkNr4ucEScRAkc9AKCnwraT9DztjAConsyuBZ7wDs/bJACgyrWR
e2zcwlIffPxhTKfFJWm3T3A=
=qDyJ
-----END PGP SIGNATURE-----



More information about the NANOG mailing list