September Backbone Engineering Report

Fri Oct 9 03:04:13 UTC 1992

Hi. This will appear in the Internet Monthly Report from ISI. Happy
reading.
	Mark

                ANSNET/NSFNET Backbone Engineering Report

                               September 1992

                Jordan Becker, ANS      Mark Knopper, Merit
                becker at ans.net          mak at merit.edu

T3 Backbone Status
==================
	During August, new RS/960 FDDI adapters were installed in 4
production ENSS nodes.	Activities are underway in preparation for
dismantling of the T1 backbone which is targeted for November. Several
steps to be completed prior to dismantling the T1 backbone include
support for OSI CLNP transport over the T3 backbone, and the
deployment of the redundant backup circuits for the T3 ENSS gateways
at each regional network.  Further activities in support of the Phase
IV upgrade to the T3 backbone are in progress.

Backbone Traffic and Routing Statistics
=======================================
	The total inbound packet count for the T1 network was
3,512,296,181, down 10.0% from August.  334,806,439 of these packets
entered from the T3 network.

	The total inbound packet count for the T3 network was
15,680,453,501 up 20.1% from August.  118,261,088 of these packets
entered from the T1 network.  

	The combined total inbound packet count for the T1 and T3
networks (less cross network traffic) was 18,739,682,155 up 13.4% from
August.

	As of September 30, the number of networks configured in the
NSFNET Policy Routing Database was 6640 for the T1 backbone, and 5920
for the T3 backbone. Of these, 1292 networks were never announced to
the T1 backbone and 1207 were never announced to the T3 backbone.  For
the T1, the maximum number of networks announced to the backbone
during the month (from samples collected every 15 minutes) was 5068;
on the T3 the maximum number of announced networks was 4458.  Average
announced networks on 9/30 were 5034 to T1, and 4428 to T3.

New FDDI Interface Adapter for ENSS Nodes 
========================================= 
	We have installed new RS960 FDDI adapters for the RS/6000
router on ENSS128 (Palo Alto), ENSS135 (San Diego), ENSS129
(Champaign), and ENSS132 (Pittsburgh). These adapters provide improved
reliability and performance.

	We are making adjustments to the Maximum Transmission Unit
(MTU) on most T3 network interfaces to support this deployment of FDDI
adapters on selected ENSS nodes.  The goal is to maximize performance
for applications designed to exploit T3/FDDI bandwidth, while
maintaining satisfactory performance for sites that still interconnect
to the T3 routers via an ethernet interface.  The MTU on all internal
T3 interfaces will initially be set to 2900, and will remain at 1500
on all ethernet interfaces.  The reason for this is to minimize packet
fragmentation for traffic that is transiting the T3 backbone via FDDI
interfaces on each end, as well as for traffic that will transit an
FDDI on one end and an ethernet interface on the other. We are
actively developing plans for subsequent FDDI deployments on
additional ENSS nodes.

Phase-IV T3 Network Upgrade Status
==================================
     The upgrades to the T3 backbone discussed in the August
engineering report have started according to schedule and will complete
prior to the dismantling of the T1 backbone.  Activities completed so
far include:

1)	T3 ENSS FDDI interface upgrades to the new RS/960 card.  This
	has been completed at ENSS128, ENSS135, ENSS132, and ENSS129.

2)	T3 DSU PROM upgrades.  We are in the process of installing new
	T3 DSU firmware in all network DSUs.  This supports additional
	SNMP function and fixes a few non-severe bugs.  The testing of
	the PROMs were completed in September, and the installation of
	these PROMs will be completed by late October.

3)	The CNSS routers in the Washington D.C. area (CNSS56, CNSS57,
	CNSS58, CNSS59, CNSS60) were successfully re-located to an MCI
	POP in downtown Washington D.C. on 9/12 for closer proximity
	to several ENSS locations.

4)	The installation of three new CNSS nodes (CNSS104, CNSS105,
	CNSS107, CNSS108) in the Atlanta MCI POP was completed on
	9/26.  The T3 circuit between Greensboro (CNSS73) and Georgia
	Tech (ENSS138) was rehomed to the to Atlanta POP (CNSS105).

T3 Link Metrics Changed
=======================
	The link metrics on the interior T3 backbone links have been
adjusted to approximate the latency on the physical T3 links, as was
discussed in the August '92 engineering report.  The new metrics
result in more east<->west traffic traversing the northern T3 route
(New York-Cleveland-Chicago-San Francisco) than the southern route
(New York-Washington-Greensboro-Atlanta-Houston-Los Angeles-San
Francisco).  Also, there are no longer any equal cost redundant routes
inside the T3 network.  This avoids assymetric internal T3 backbone
routing.

Dismantling the T1 Backbone
===========================
	The current target for dismantling the T1 backbone is November
'92.  This will occur after (1) the remaining networks using the T1
backbone have been cut over to the T3 backbone (these are: ESnet,
EASInet and CA*net); (2) the OSI CLNP transport capability over the T3
backbone is in place; (3) the T3 ENSS nodes are backed up by additional
T1 circuits terminating at alternate backbone POPs. These activities
are described below.  

Remaining Network Cutovers
--------------------------
	The ESnet cutover is waiting for a new version of software to
be configured for the ESnet router peers at FIX-West and FIX-East.  A
routing plan for this cutover has been developed with ESnet and
Argonne National Lab staff.

	A new T1 ENSS will be installed in CERN, Switzerland to
provide connectivity to the T3 backbone for EASInet.  Cutover of
EASInet traffic will occur when this installation is complete.

	The Seattle RT E-PSP for CA*net is being converted to run
the CA*net software and operate as part of CA*net's domain. It will
run gated and speak BGP with the T3 ENSS. Once this has been debugged
and tested the Princeton and Ithaca connections will similarly be
upgraded. 

OSI Support Plan
----------------
	We have begun to test the RT/PC OSI encapsulator software that
was described in the August '92 engineering report.  We plan to
continue to run this over the T3 test network during early October,
and begin deployment of the encapsulator software on production
RT-EPSP nodes in late October.  Native CLNP switching services will be
available in a future release of the RS/6000 system software which is
scheduled for deployment on the T3 network in early 1993.

T1 ENSS Backup Circuits
-----------------------
	The installation of new T1 circuits between the existing T3
ENSS nodes, and a backup CNSS node is being scheduled to begin in late
October.  The initial topology for T3 ENSS backup is illustrated in a
postscript map that is available via anonymous FTP on ftp.ans.net in
the file /pub/info/t3enss-backup.ps .  We are in contact with several
regionals to schedule the installation of these backup links.  We will
discuss specific backup routing plans and other details with each
regional.

Increase Routing Table Sizes on T3 Network
==========================================
     We continue to experience an increase in ANSNET/NSFNET advertised
networks, (see Backbone Traffic and Routing Statistics, above) The
current on-card routing table size on the T3 router RS960 card
(T3/FDDI) and T960 card (T1/ethernet) supports 6,000 destination
networks with up to 4 alternate routes per destination.  The current
on-card routing tables are managing on the order of 12K routes
(including alternate routes to the same destination).

     We are now testing new adapter microcode for the RS960 and T960
interfaces that will be deployed shortly on the T3 backbone that
supports up to 10,000 destination networks with up to 4 alternate
routes per destination network.

     We also continue to work on support for on-card route caching
which will significantly increase the upper limit on the number of
routes to unique destination networks that may be supported on the T3
network.  This software will be available with the AIX 3.2 operating
system release of the RS/6000 router software in early 1Q93.

Notable Outages in September
============================

MCI Maryland Fiber Outage - 9/13
-------------------------                           
	On Sunday evening 9/13 there was a major MCI fiber outage on
the east coast that affected T3 circuits between New
York<->Washington, and New York<->Cleveland.  The same outage also
caused several T1 safetynet circuits to go down.  The fiber outage was
cleared early Monday morning 9/14.  During the outage, we had 4
circuits on MCI restoration facilities that were moved over to
temporary restoration facilities and then moved back after the outage:

New York <-> Cleveland T3, and T1 Safetynet
St. Louis <-> Lafayette T1 Purdue
Austell <-> Pittsburugh T1 Backbone

During the fiber outage, some users may have experienced performance
problems due to alternate routing over safetynet circuits, but there
were did not appear to be any T3/T1 ENSS/NSS nodes that were isolated.

CNSS32 Outage - 9/14
-------------
	Following the fiber outage described above, there was an
unrelated equipment problem that occured on the New York T3 backbone
router (CNSS32).  This problem started on Monday morning and was
ultimately resolved by replacing the RS/6000 I/O planar board hardware
on Tuesday evening.  Traffic was routed around CNSS32 during this
outage with the exception of the period 12:30-13:30 EST on 9/15 during
which the hardware failure induced a network segmentation causing
several ENSS nodes to be isolated.

AIX 3.1 TCP Bug
---------------
	We have identified a bug in the AIX 3.1 system software that
can leave RS/6000 router TCP sessions (e.g. external BGP) in a
deadlock state.  We have observed some isolated problems caused by
this bug on ENSS134, and ENSS136 in September.  We are working to
isolate and correct this problem.