Preview of ANSNET/NSFNET Backbone Engineering Report

Fri Feb 28 23:53:07 UTC 1992

We will include a more detailed report in the February Internet
Monthly Report, but this summary gives a general report of
events on the backbones over the last month. 
	Mark


ANSNET/NSFNET Backbone Engineering Report 
Preview of February, 1992 End-Of-Month Report 
=============================================
     Mark Knopper, Merit 
     Jordan Becker, ANS 

 
Summary
======= 
     The T3 network continues to perform extremely well as it has since
last November.  During February, we have cut over a significant load of
traffic from the T1 backbone to the T3 backbone.  Midlevel networks that
were cutover to use T3 in February include SURAnet (at both College
Park and Atlanta), NyserNet/PSI, San Diego (SDSC and CERFNet), and
SesquiNet.  These are in addition to the other midlevel & regional
networks that have previously been cut over to use the T3 system.  We
are coordinating with several other midlevel networks that we plan to
cutover to T3 during the month of March.

     The midlevel networks continue to peer with both the T1 and T3
networks, and continue to use the T1 backbone to communicate with sites
that have not yet cut over to the T3 backbone.   This minimizes the load
on the T1/T3 interconnect gateways in Ann Arbor, Houston, and San
Diego.  The interconnect gateway load has decreased as we have added
load to the T3 system.  We are also in the process of installing a 4th
interconnect gateway at Princeton.


T3 Network Status
=================
     Performance on the T3 network has been very good.  There have
been reports of intermittent packet loss, but this has been isolated to
sources outside of the T3/T1 systems.  As we migrate additional traffic
onto the T3 network we are collecting daily reports on peak traffic load,
and any packet discards on all CNSS/ENSS nodes in order to detect any
problems caused by the added load.  We are now measuring daily
average sustained loads of about 5Mbps across a typical CNSS node (all
interfaces) with an average packet size of about 200 bytes.

     The T3 backbone continues to be very reliable.  There was a
hardware problem on the backbone CNSS machine at Cleveland that
resulted in two outages.  We also continue to experience an occasional
black link on CNSS-CNSS links.  With the redundancy in the T3 network,
none of these problems resulted in extended outages for any end users. 
There was also a routing software problem with the flooding of external
BGP updates between the IBM and Cisco routers which we have worked
around, and an SNMP monitoring problem which was identified and
corrected. 


T1 Network Status Summary
=========================
     Performance and reliability on the T1 backbone has not been as
good as T3 during February, but this is improving as we have (1) cut
significant traffic over to the T3 system (2) are fixing several chronic
problems that have been isolated and focused on during the last month.
Some of these chronic problems and fixes include: 

-    Congestion at NSS10 has been minimized by cutover of
     PSI/NyserNet traffic to the T3 system.  We also replaced the PSP-
     10-16 node to ensure that we were not having any intermittent
     hardware problems.

-    We recorded an above average number of T1 circuit outages
     reported during February.  The increase in outages was largely due
     to two unrelated fiber cuts in the MCI network during the week of
     2/17.

-    The "DCD Waffle" problem reported previously has been further
     diagnosed as two separate problems, and software corrections to
     these problems have already been partially deployed with the
     remaining software updates being tested on the research network for
     deployment during the coming week.  A detailed description of these
     problems will be documented in the February Internet Monthly report.

-    An intermittent PSP crash problem is still being investigated.  We
     suspect that this involves an RT kernel bug in the virtual memory
     system.

-    The fix to the "internally generated" ICMP net unreachable message
     problem has been successfully tested on the research network and
     will be deployed on the production network during the coming week. 
     This should serve to minimize host problems resulting from any
     routing instabilities in the T1 network.

-    A new interior routing problem was identified where IS-IS link state
     PDUs were being truncated at sites where the regional was
     announcing over 2000 networks to the T1 backbone.  A patch to the
     routing software to allow up to 2200 networks announced to a single
     NSS was already deployed as a near term fix.  A new routing
     daemon that compresses the encoding of networks in the IS-IS link
     state packet is now being tested for deployment as a longer term
     solution.