Palo Alto Power Hit

Sean Donelan SEAN at SDG.DRA.COM
Tue Feb 9 20:47:25 UTC 1999

Apparently my normal sources inside the carriers got stomped on by
senior management, and aren't allowed to talk to me anymore.  So here
is a summary of unofficial reports from various unofficial sources.

The West Coast is experience severe weather, including storms and
flash flooding.  There are multiple weather advisories up and down
most of the coast.  Approximately 55,000 people are without power.
At the moment, no reported deaths due to the storms.

Internet related incidents

There were power problems through the early morning hours.  A
major power surge hit the Palo Alto grid at 7:16am PST.  I haven't
seen any SCADA information on just how large the surge was.  This
appears to be the instigating factor for several carrier outages.

Several significant Internet trunks pass through the Palo Alto

Customers of several carriers reported problems including
MCI/Worldcom/UUNET/MFS, GTE/BBN, and Sprint.  One person
reported loss of residential phone service, which may indicate
Pacific Bell was also affected. A lot of dead routes, and
partially routed CIDR blocks seem to indicate those backbones
may have been partitioned for some time.

UUNET and MFS customers reported loss of a Palo Alto node.  I don't
know if this referred to their equipment in the PAIX building, or at
another Palo Alto location.  MFS had a power problem at PAIX a few
years ago, but elected to use utility power instead of PAIX power.
PAIX reported their power equipment absorbed the surge, and continues
to operate normally.  BBN at Stanford appeared to have some communication
problems, no word if any rats were implicated.  MAE-West-San Jose,
Mae-West-Ames, and PacBell AADS appear unaffected.  Sprint didn't
appear to have any circuits down, but had enough problems with the
few people they peer with externally, that Internet routing wasn't
working well.

A common report was the power surge either blew the fuses, or
the entire power supply on a MFS OC192 piece of equipment.  This
may be the source for some reports of a fiber cut in the area.
Most problems cleared up around 9:15PST after the fuse/equipment
was repaired.
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
  Affiliation given for identification not representation

More information about the NANOG mailing list