BBN (GTE) Suffers another major power problem.
jhawk at bbnplanet.com
Fri Aug 8 18:15:25 UTC 1997
> >If you're in a 24x7, customer-driven market, you should have either:
> >a) enough go-juice (via fuel contractor) to go until the grid is on-line
> Did you read the previous email from jhawk? There is an
> extremely large generator at one of the affected facilities
> (read: cogeneration plant) but no amount of fuel helps if
> the plant is damaged in the explosion.
Unfortunately, the previous mail from jhawk went to BBN's local
NANOG exploder and not to the public NANOG list :-(
As long as I'm taking a point to clarify publically, I'll try to be complete.
> p.s. No, we've have not deployed redundant power plants (yet)
BBN has a number of facilities, some of which are colocated in telco
facilities, and some of which are not. Those that are in telco facilities
have (as you would, of course, expect) hours-to-days of backup power
in the event of a catastrophic failure.
Those that are not, however, do not.
BBN has major facilities colocated with both MIT and Stanford, both of
which have co-generation power, which is expected to be available
should utility power failure.
Of course, it is well-recognized, and recent history serves to enforce that
quite well, that this doesn't help for many kinds of failures, especially
if the distance between the backup power source and your equipment
is large, with multiple possible points of failure.
Sean Doran is perfectly correct when he points out that market economics
are central here. When engineering redundancy, there's going to always be
some number of customers that you are willing to let suffer in the event
of a catastrophy. In some cases, maybe that number is zero. In other cases,
perhaps it's some number of percentage points. It clearly doesn't make sense
to engineer for the case where a disaster takes out hundreds of customers,
leaving only one remaining :-) I'm not prepared to address what our outlook
on this is, of course, however we generally feel that facilities with
co-generation are less likely to be in horrible power positions than other
facilities. Maybe not "less by a lot", but certainly "less".
Unlike the Stanford case, however, while we did not have on-site generator
power, we were certainly more prepared for it, and it in fact arrived,
largely coincident with the return of utility power. That's the way it goes.
Attached are some exerpts of summaries from the MIT which you might
care for if you want nitty-gritty details.
Folks on nanog who commented that having an on-site generator for outages
being the way to go are certainly correct -- if your marketting economics
justify them. I can't really speak to ours or our future plans, however.
To answer some other questions, BBN does indeed have multiple POPs in the
Cambridge/Boston area, and this only affected one of them.
The suggestino that it was "physically impractical" to power the equipment
in question (Chris Masto) is simply incorrect.
Matt Ringel's comment that the power was out from 1730 through 2245 (EDT)
is not completely correct (see below), as there was quite a long period
of restored power in the middle, and the second outage was more widespread
than the first. Note, also, that UPS power caused our equipment to run for
quite a while longer than the numbers below indicate, but I don't have
precise figures on that handy.
The assertion that we had emergency generator backup at this location
(Eric Osborne) was incorrect; a generator was brought in as the outage
drew to a close, and co-generation was normally expected to be
The initial mail (Patrick Chicas) implied that power to the building
housing our POP was lost -- in reality the problem was roughly
half of the city of Cambridge, MA.
I think that just about covers all the bases.
Subject: Power Outage August 7 , 1997
Date: Fri, 8 Aug 1997 07:39:43 -0400
At approximately 5:35pm on August 7, 1997 a manhole explosion in the
Kendall Sq area caused Cambridge Electric to loose Kendall Station. MIT
lost all power and the gas turbine tripped. Power was fully restored at
At approximately 7:05pm, a second manhole explosion,caused 1 fatality,
and injurys to 4 other Cambridge Electric utilitymen including a
Cambridge Policeman. Putnam Station was also tripped and MIT lost all
At approximately 10:30pm, MIT had power restored to all buildings within
our distribution system. Several East campus ( E28, E32, E42, E56, E60,
NE43)and North/Northwest buildings( N42, N51/52, N57, NW10, NW12, NW14,
NW15, NW17, NW20, NW21, NW22, NW30, NW61, NW62, W11, WW15) which are fed
directly from Cambridge Electric, were restored by Cambridge Electric
Cambridge Electric is still sorting out the chain of events. At last
discussions with them, a total of 3 manhole explosions had taken place.
Additional information will be posted when available.
Subject: Power Problem Update
Date: Fri, 8 Aug 1997 09:31:05 -0400
Friday morning, @ 0930
Power problem update:
The MIT cogeneration facility was damaged during the power problems that
occurred last night. As a result, the campus is operating totally on
power supplied by Cambridge Electric.
At this time, Cambridge Electric is stable, however, due to the events
that occurred yesterday and early this morning, it is uncertain how
reliable their distribution system will be until repairs are
MIT is working as fast as possible to get repair parts for the
cogenerator and get it on line. It is expected that the parts will
arrive this afternoon for immediate installation.
More information about the NANOG