Looking for advice on datacenter electrical/generator

Fri Apr 4 02:59:16 UTC 2003

On Thu, 3 Apr 2003, Sean Donelan wrote:

> > That was 25 Broadway (Telehouse USA).  The air intake filter for their
> > generator clogged from all the WTC dust in the air.  Pretty rare
> > circumstance...
>
> There were some initial reports that this was the case, but I have
> never seen the final report from Telehouse what the final analysis
> found.  If you remember there were also reports about a water pump
> failure, raditor clogging, etc.

I know the clogged air filter was what we heard for 32 Old Slip.  Since
the telcos in that building (ATT, Focal, GNaps) only operate their own
battery strings and not generators, they relied on the building generator.
They didn't have any pull in getting maintenance folks back into the area
to work on the genset.  When power was restored initially it was not
enough to handle cooling, so stepping out onto the 5th floor landing was
like entering a blast furnace.

At Telehouse, this is what I dug out of my archives, presumably from John
Shields.  Some things to note are that the initial failure was due to the
facility running out of fuel.  Ooops.  Once refueled, the water pump
supposedly died.  If I recall correctly there were two other gensets
running for the 6th floor, which at the time was basically unoccupied.  It
would have been a better design perhaps to tie both floors together and
get N+1 redundancy on the gensets.  But Telehouse has a poor track record
with whomever does their electrical work.  Before we moved to 25Bway, we
went through a period where a transfer switch would fail during tests at
33 Whitehall.  If I recall correctly this happened at least 5 times in one
year.

-----

To our 25 Broadway Clients:

We'd like to bring you up to date regarding the power situation at 25
Broadway.

As you are aware, the commercial power supply to 25 Broadway was lost at
about 16:35 EDT Tuesday, September 11th, following the terrorist attack on
NYC. With the cooperation of the NY City Office of Emergency Management
and many others, we were able to replenish the diesel fuel. At
approximately 15:30 Thursday, however, our critical power generator's
water pump failed and power was lost to most of our customers' equipment
on the 5th floor.

We have aggressively pursued a two-pronged approach to
restoring power:

1) TELEHOUSE prevailed upon Con Edison, the regional commercial electric
power supplier, to deliver a portable generator to the site. The generator
will be set up in the street adjacent to the building and connected by
their electricians through a transformer than Con Ed delivered earlier
today and wired directly into the building's bus. We expect to have power
restored to your equipment between 15:30-17:00 EDT.

2) Although we were originally advised that we would have a new water pump
shipped to the site for installation this morning, Caterpillar's
representative in this region (H.O. Penn) was unable to locate one in the
metropolitan area. We have arranged to have a pump forwarded from the
warehouse in Pennsylvania via expedited shipping. The pump is expected to
arrive in NYC by approximately 20:00 tonight with installation being
planned for tomorrow. This will make available an additional level of
backup beyond the generator provided by Con Edison.

We understand that you have endured hardships as a result of this problem.
We sincerely appreciate your patience and understanding as we, along with
the rest of New York, work to overcome the stresses that this catastrophe
has forced upon all. Our Customer Assistance Center (CAC) is staffed 24/7.
Please feel free to contact us at [Confidential - Removed] at any time you
have questions. Our email address [Confidential - Removed] should also be
working again soon after the restoration of power.