Network Outage

Fri Mar 22 01:38:57 UTC 2002

Well, I used to be a contractor for MCI/WCOM, so I know all the numbers to 
call and the names of people to ask for. They as a rule are very mysterious 
about anything outage related. The first rule to get better service, is 
always say OK to intrusive testing. I've had jacks go bad, and the bridge 
point isn't a wholesome indicator and the fault can be 2 inches away. 
Besides, I was hard down, all zeros, so it didn't matter.
After some arm twisting at Bell Canada, they confirmed it wasn't on their 
side of the ditch, so I started calling people I know here. The usual 
characters at the WCOM NOC were calling and leaving useless messages on my 
answering machine, but I called Sardinia, and got them to check outbound, 
towards Canada, and they said the T3 looked wholesome, but he couldn't 
access it at T1 level. I did however, find my Ticket appended to some 
others, so I knew it had to be at a bigger pipe than my T1. I know where 
they can check at T1 level, at 3DE, (the old MFS site, where I used to 
work) so I called, and a girl I worked with told me they had a DS3 
cross-connect cable failure earlier, and it was repaired. I called over to 
the transport desk, and they confirmed the circuit was outbound twords 
Canada. I put two and two together and when I called the NOC back (they 
hadn't called my pager as instructed when the cut was cleared) they said 
the problem had been cleared and the circuit restored. Turns out, the NYC 
stuff was independent of the Canada problem, (which was my top priority) 
and the NYC stuff cleared on its own about 15 mins after I talked to 
Sardinia. Never got a positive cause on that one, but its gone, so its ok 
by me. Theyre stretched so thin on manpower now, and rumors are flying 
theres to be another round of layoffs next week, groan.

Anyway, all the circuits are back and wholesome, so Im happy...back up at 
4:10pm

At 19:59 3/21/02 -0500, you wrote:
>What did you end up getting from them.  I lost some ccts out of 60 Hudson
>and they would not cough up to a problem, only hour later during a co-op
>test did the problem "clear on its own".  Any details you can provide would
>be great.
>
>
>----- Original Message -----
>From: "blitz" <blitz at macronet.net>
>To: <nanog at merit.edu>
>Sent: Thursday, March 21, 2002 3:00 PM
>Subject: Network Outage
>
>
> >
> > Good day,
> >   There has been a MCI/Worldcom outage near the Sardinia, NY switching
>site.
> > I have been in contact with them since around 12:00 EST and they are
> > presently trying to localize the problem.
> > I've lost a couple international circuits to Canada, and some to NYC. I
> > don't know if anyone else is experiencing similar problems.
> >
> > This is just a heads up for the list (and my first post..hello all)
> >
> > Marc Blitz
> >