POWER: CORRECTION: San Mateo/San Francisco Power

Sean Donelan SEAN at SDG.DRA.COM
Thu Dec 10 14:29:26 UTC 1998

Some corrections:

Official Start time: 8:17AM Pacific Standard Time
Official End time: 3:52PM Pacific Standard Time

Root cause, failure by a work crew to *UN*ground two safety ground
straps after completing work on the 115Kvolt line at the San Mateo
Coyote Point substation.

1 Fatality, a women struck at an intersection with non-functioning
traffic signals.

San Francisco has two subway systems.  BART was able to move their
trains to the next station, reportedly in some cases after a 1-2 hour
delay.  MUNI evacuated people on foot through the tunnels.

San Francisco International Airport had emergency power for runway
lights and navagation, and did land planes during the outage.  The
jetways, baggage conveyers, security checkpoints, ticketing systems, etc 
did not have power.  Because the airport can't handle large numbers of
passengers without power other flights were delayed, cancelled or
diverted to other airports.

Effect on Major/Minor Internet providers.

A number of people wrote in saying that provider XYZ had problems.  There
may not have been any 'major' problems with 'major' providers, there were
some minor/major problems with major/minor providers.

How individual customers were affected varied, mostly depending on the
customers' own backup planning.  Generally the Customer Premise Equipment
lost power (or backup power) before the ISP's equipment went down.

Telephone calling volume was up 43% in the Bay area.  This may have
lead to a higher number of ISP/busy/no connection problems with dialup

The most common reported problem was with webserver farms only having
limited backup battery.  I also received a report of a provider's
customer service PBX system battery failing, so their phones gave ring-no
answer.  Providers generally gave priority to routers and access servers
over web and mail servers.  You could often use the Internet, but your
mail or web page may have been unavailable.  E-Mail is automatically
queued for 2-4 days, so very few e-mail messages should have been lost,
only delayed a few hours.

When evaluating how your provider did, ask where their POP is and if it
lost utlity power.  I've noticed some provider sales people already
enaging in some competitor bashing, saying they stayed up while 
provider X didn't.  However in some cases, it was because they were
located in a different area and never lost utility power.

Effects on other industries.

Banks may have a lot of backup systems to protect them, but their customer
access systems appear less reliable.  600 ATM machines, and 200 bank
branches in the area were shutdown due to the power outage.  One person
noted that Nordstroms, long noted for their customer service, remained
open throughout the power outage.  At other stores, clerks couldn't make
sales without computer price lookup and approval, especially for 'sale'
items.  VISA/Mastercard say their credit card approval systems remained
operational, but some store's inhouse PBX and Verifone terminals didn't
have battery backup.  Some grocery stores shutdown, others implemented
an "honor" system where people used pencil and paper to write the prices
off the shelf tags, since the UPC laser scanners require power.
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
  Affiliation given for identification not representation

More information about the NANOG mailing list