San Francisco Power Outage

Wed Jul 25 21:30:07 UTC 2007

Michael Dillon writes:
>> And the stories that the power guy I'm working with tells 
>> about foreign facilities, particularly in middle east war 
>> zones, are really scary...
> 
>> We fundamentally do not have the facilities problem 
>> completely nailed down to the point that things will never 
>> drop.  Level 4
>> datacenters can, and will, fail.   Nothing you can do including
>> just doing 48V DC for everything are truly foolproof solutions.
>
>A single level 4 datacenter is a Single Point of Failure!
>
>Two of those middle-eastern style facilities is... ?
>Has anyone actually kept track of all these data center failures over
>the years and done some statistical analysis on it? Maybe two half-baked
>data centers is better than one over the long run?
>
>Remember that one 10-12 years ago in (Palo Alto, Mountainview?) where a
>lady in a car caused a backhoe driver to move out of the way which
>resulted in him cutting a gas line which resulted in the fire department
>evacuating the data center, cutting off electricity in the area, and
>forbidding the diesel generators to be switched on? 

Santa Clara.

I was working right outside the evacuation radius.

Which exchange point was in the building?  PB-NAP?  CIX?  I remember
we had a net-dark event associated, but not which one.

It was a bad day...

The lesson, as you point out, is that geographical redundancy
is sometimes necessary.  This is as true for providers as for
datacenter end-users...  

-george william herbert
gherbert at retro.com