Newedge Networks meltdown?

Peering Peering at xspedius.com
Tue Nov 9 20:52:36 UTC 2004


I can understand not wanting to get Alcatel support...we use Alcatel's
NMS too and it's pricey for everything, support, upgrades, etc. (thank
goodness it's now on the edge of our network and not on the core...hail
Juniper!!).  But you gotta do what you gotta do.  I can't imagine us
being down that hard that long before we got the vendor involved.

Diane Turley
Network Engineer
Xspedius Communications Co.
636-625-7178


-----Original Message-----
From: owner-nanog at merit.edu [mailto:owner-nanog at merit.edu] On Behalf Of
netadm
Sent: Tuesday, November 09, 2004 2:32 PM
To: nanog at merit.edu
Subject: RE: Newedge Networks meltdown?





Statement from New Edge to Customers:
-------------------------------------

Edger Customers and Partners, 

We are experiencing major network issues with our Alcatel Network
Management System (NMS).  Early last evening we had degradation in our
primary NMS and we switched to our secondary backup NMS as per standard
procedure.    We then experienced similar degradation difficulties with
our backup NMS.   We have worked through the night on this and we have
brought in Alcatel.

Much of our network is stable. We estimate about 30% of our permanent
virtual circuits (PVC) for customers may have been impacted.  Customer
affecting symptoms can take the form of performance degradation on a
circuit or an actual outage in the form of failure to sync at Layer 3.  

We are now back on our primary NMS and we believe we have this situation
stabilized.  We are rebuilding our customer PVC's and the situation
should improve throughout the day.  We have added extra resources in NOC
and our call centers.

Our Service Level Agreements (SLA) call for 99.5 % up time and we have
always far surpassed these SLA targets. This is the first outage of its
type in the five year history of New Edge Networks. 

We are deeply embarrassed by this network outage and we apologize to our
customers.   We will do everything possible to fix this and we will
communicate on the hour as we do so.

Dan Moffat 
CEO New Edge Networks 



-----Original Message-----
From: Bubba Parker [mailto:sysadmin at citynetwireless.net] 
Sent: Tuesday, November 09, 2004 11:10 AM
To: Mark Radabaugh
Cc: nanog at merit.edu
Subject: Re: Newedge Networks meltdown?



I had a DS3 go down with them last night, they said that a lot of their
PVCs all showed active must had to be bounced to continue working, and
they'd only know which ones were down if people called and told so.

On Tue, Nov 09, 2004 at 07:40:36AM -0500, Mark Radabaugh wrote:
> 
> Seems there is a major problem within NewEdge networks this morning. 
> From the recording on the NOC line it sounds like they lost an
internal 
> database and PVC's at many of the switches.
> 
> Anyone know what happened?  Let the speculation begin :-)
> 
> Mark Radabaugh
> Amplex
> 

-- 
Bubba Parker
sysadmin at citynetwireless.net
CityNet LLC
http://www.citynetinfo.com/



More information about the NANOG mailing list