FINAL: MFS Outage: 6:30PM PDT 4/18/97 - ~1:30AM 4/19/97

Sun Apr 20 12:14:32 UTC 1997

Well done.

-- 
Rusty

---------------------------------------------------------------------
|  MCI Communications                                               |
|  Internet MCI                                   rusty at mci.net     |
---------------------| http://infopage.mci.net |---------------------

> This final incident report is being sent at 1:14AM 4/19/97:
> 
> UPDATE:  (7:30PM PDT)
> 
> Starting at approximately 6:30PM PDT, MFS's power
> unit on their fiber mux located at the Digital Internet
> Exchange is resetting every 30 seconds to 1 minute.
> 
> All MFS fiber facilities are down during the power
> reset.  This includes all MFS/Worldcom DS3 and T1
> circuits entering the PAIX collocate facility.
> 
> The IX Operations staff (Digital Internet Exchange
> Operations) believes that the problem is due to a
> bad rectifier on the fiber mux's power rack.
> 
> We are currently waiting for MFS to provide a dispatch
> to fix this problem.
> 
> UPDATE: (11:00PM PDT)
> 
> At approximately 7:30PM, MFS arrived on site.
> We offered them clean power from our in-house
> UPS and rectifiers at this time, which would have
> prevented the rapid, regular resetting of their
> line cards.  They were not interested in the power
> at this time.
> 
> By 8:30PM, they were convinced their rectifier
> was the cause of their power loss.  They still are
> not interested in our power.
> 
> By 10:15, they had restored power to their rectifier.
> Many of their line cards, including an OC48, did not
> return back to an operational state, likely due to
> the rapid resetting previously.  Again we offered our
> power, however now they definitely were convinced it
> was unnecessary as their rectifiers were functioning.
> Instead, they are forcing us to wait until next week.
> 
> As of 11:00, the line cards are still not functioning.
> 
> UPDATE: 12:10AM 4/19/97
> 
> As of 11:45PM, the MFS technicians decided to go
> ahead and begin cut over to our power so that at
> least the power is stable while they work on the
> line cards.  They are currently working on this cutover.
> 
> UPDATE: 1:14AM 4/19/97
> 
> By 12:15AM they changed their minds on the power 
> cutover again.  The electricians installed two
> redundant rectifiers and two additional batteries
> to the system, which should provide power until
> the cutover to the Digital Internet Exchange power
> next week.  Also by this point they have determined
> that the problem with the line cards is due to the
> fact that they lost their configuration during the
> power fluctuations.  They begin reprogramming them
> at this time.
> 
> At 12:35AM, MFS customers begin reporting their
> services returning to an operational state.
> 
> At 1:00AM, the MFS representative completes all
> programming.  He says that the circuits will return
> to service, one by one, using a "time lapse" system
> over the next 30 to 35 minutes.  By 1:35AM, all
> services should be restored.
> 
> At 1:25AM, MFS left the site saying all circuits 
> that were "called in" by their customers were on-line.
> 
> If by 2:35AM your services are not restored, please
> contact MFS and give them your circuit I.D.  We are
> available for any remote hands requirements in the
> facility at 1-800-209-9898.  
> Thank you,
> 
> -Jay Adelson
> 
> -----------------------------------------------------------
> Jay Adelson                  |    Email: adelson at pa.dec.com
> Operations Manager           |    Voice: (415) 617-3607
> Digital Internet Exchange    |      Fax: (415) 617-3233
> -----------------------------------------------------------
>                Secure. Reliable. Connected.
> 