Equinox Power Failure

Fri Jul 1 00:38:17 UTC 2005

On Thu, Jun 30, 2005 at 03:52:00PM -0700, Pablo's Gmail wrote:
> 
> Does anyone have related and/or specific outage information to the
> Equinox IBX facility in Chicago that occurred this past weekend? I'm
> more curious of actual root cause of their power facilities, outage
> timelines, response times...etc.

Equinox outage in Chicago? Did the power to the treadmill go off while you 
were working out? :)

Equinix actually did a pretty good job of providing detailed information 
about what went wrong and why (after the fact), and kept customers up to 
date regarding the resolutions and testing activity afterwards. If you are 
a customer you should have gotten a copy of the report by now. I'm not 
going to paste the actual contents of it here (since it has one of those 
pesky industry-standard confidential information tags on it), but you can 
probably get someone at Equinix to give you a copy.

There are really only two things I can fault Equinix for regarding the 
handling of the incident:

#1 They didn't get an announcement of the situation out until 4 hours 
after the failure.

#2 They should have had better monitoring systems in place to realize that 
they were putting out dirty power, and taken action to shut it off rather 
than putting out 90V on 120V AC circuits and bouncing on/off every minute 
until utility power was restored. Nearly everyone I've talked to lost some 
equipment during this, in some cases millions of dollars worth of gear 
(thank god for insurance :P). The recovery process from the damaged 
hardware lasted much longer than the actual outage, at least for us and 
our vendors.

Other than that, they did the best they could with the situation, followed 
up with corrective action and useful/detailed information after the 
incident was over, and actually delivered on SLA credits much faster than 
other colos in similar situations.

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)