December Backbone Engineering Report

Mon Jan 11 06:16:21 UTC 1993

Hi. This report will appear in this month's Internet Monthly Report.
	Mark

              ANSNET/NSFNET Backbone Engineering Report   

                             December 1992 

                Jordan Becker, ANS     Mark Knopper, Merit   
                becker at ans.net         mak at merit.edu   

Network Status Summary 
====================== 
     Following the T1 backbone disconnection on Dec. 2nd, the T3 is
now supporting all NSFNET services.  A 2nd ethernet interface was
installed in ENSS136 to support interim connectivity to the MAE-East
facility.  The former T1 backbone class B address (129.140) was returned
to the NIC.  CA*Net peer router hardware reconfigurations will be
completed in January to allow direct T3 network peering.  A plan has
been developed to dismantle and remove the RT routers at regional sites.

     Following the software changes resulting from instabilities observed
during the MBONE multicasts in November, the T3 backbone routing
stability has been very good in December.  Daily reports are now
generated on internal and external peer network routing stability.

     Seven new RS960 FDDI interfaces were installed on ENSS nodes in
December.  Reliability and performance of the FDDI hardware has been
very good.

Backbone Traffic and Routing Statistics   
=======================================   
     The total inbound packet count for the T3 Backbone (measured using
SNMP interface counters) was 22,009,352,089 up 5.0% from November.

     As of December 31, the number of networks configured in the NSFNET
Policy Routing Database was 8561 for the T3 backbone. Of these, 1760
were never announced to the T3 backbone.  The maximum number of
networks announced to the T3 backbone during the month (from samples
collected every 15 minutes) was 6187.  Average announced networks on
12/31 were 6154.

     Previously when the T1 Backbone was operational the overall
average number of networks announced via the primary configured AS
path was around 88%. Since the T1 Backbone was dismantled this
average has gone to around 95%. Graphs of this information are 
available for anonymous ftp on merit.edu, in pub/nsfnet/offnet,
as postscript files. 

Routing Software and Stability on the T3 Network   
================================================
     Higher than usual MBONE activity in November resulted in exterior
routing instabilities and routing instability internal to ANSnet.  Following 
the immediate corrective actions that were taken in November, a number
of software changes were made in December.  Changes to the RS960
microcode and efficiency improvements in the routing software have
resulted in improved ANSnet internal routing stability (regardless of the
presence of external route flapping).  The MBONE participants have made
changes to route MBONE traffic over the T3 backbone whenever possible,
resulting in improved routing stability within certain US regional networks. 

     Detailed reports of internal ANSnet and external peer network routing
instabilities are now generated on a daily basis.  The ANSnet internal
report attempts to summarize outages based on number and duration of
the IBGP disconnects.  The program that generates this report looks at
IBGP disconnects in a short time window.  It tries to assign the outage to
the node with the most disconnects, and then eliminate all disconnects on
other machines associated with that router.  If after eliminating disconnects
related to the router assigned to the outage, more disconnects remain in
the same time window, the process is repeated, assigning secondary and
tertiary responsibility for the outage and so on until all disconnects are
accounted for.  The time window is extended to cover all disconnects that
are not separated by more than a five minute period in which there is no
change to internal routing in the network. 

     Using this method, data sufficient to generate reports on internal
ANSnet routing stability has been collected continuously since October '92. 
Over the course of December '92, all but two ANSnet nodes reported
99.0% internal routing stability (no changes to internal routing).  These two
nodes reported 98.9% stability, mostly due to a problem early in
December.  Most nodes reported 99.0% to 99.5% stability.  Two reported
better than 99.5% stability.  The vast majority of this instability is due to
the regular configuration updates and scheduled maintenance.  With the
dismantling of the T1 backbone, activities are underway to reduce the
duration of time required to process the bi-weekly router configuration
runs, and scheduled maintenance.

     Similar data on external routing stability has been collected
continuously since December '92.  The external routing stability reports will
record an external route which is withdrawn from the ANSnet due to: (1)
an EGP route timing out (not being advertised by an external peer), (2)
an explicit BGP unreachable received from an external peer, (3) the loss
of an external EGP or BGP session that is not accompanied by loss of
internal IBGP sessions.  Excluded are the following cases: (1) we do not
observe changes in the next hop at the ENSS, (2) the route is replaced
by another route on the same ENSS, (3) a route is lost and backed up
by another route on the same ENSS, (4) connectivity is lost due to
ANSnet internal problems (e.g. configuration runs or other).  Some of this
data has already been shared with specific regional networks to get
feedback.  In January, these reports will be fully automated and distributed
to peer networks which are advertising networks that are withdrawing
reachability excessively (e.g. exhibiting great instability). 

     After the recent FIX-East reconfiguration, a bug in rcp_routed was
observed that would result in an ENSS installing a route locally that had
previously been advertsized by a peer, where the route was rejected on
the ENSS due to policy constraints (no NACR request).  Such a route
would incorrectly be installed on the ENSS (but not advertised) when the
destination becomes completely unreachable.  On ENSS136, the existence
of multiple peers coupled with the use of default routing by some peers
could form a routing loop when this occurs.  This problem in the
rcp_routed software has been corrected and is deployed on ENSS136. 
Deployment of the same changes across the rest of the ANSnet is
scheduled for early January.

     Another planned change to rcp_routed in January involves fixing the
metric_in function for EGP.  This will allow regional peer routers to be
able to have an EGP session with the same ENSS on two different
interfaces (FDDI and ethernet) and prefer the FDDI via metrics. 

RS960 FDDI Deployment Status  
============================  
     During the month of December, we installed RS960 FDDI adapters
at the following 7 locations: Boston (E134), Argonne (E130), FIX-E (E145),
Houston (E139), Ithaca (E133), FIX-W (E144), Seattle (E143).

Domain Name Service for ANSnet 
============================== 
     In December, ANS made two adjustments to the DNS administration
of the ANS backbone.  First, the "t3.nsf.net" domain was renamed to
"t3.ans.net", and second, we added geographical designators to all CNSS
nodes.  ENSS names were not changed other than in being moved to the
"t3.ans.net" domain. The following excerpt of a traceroute from ANS
Elmsford to Merit in Ann Arbor illustrates the ANSnet DNS conventions. 

3  t1-2.New-York-cnss35.t3.ans.net (140.222.35.4)  7 ms  7 ms  7 ms 

     ANS Elmsford's ENSS is attached to T1 Port #2 on CNSS35 (a T1
     concentrator in the New York POP). 

5  t3-3.New-York-cnss32.t3.ans.net (140.222.32.4)  8 ms  8 ms  8 ms 
6  t3-1.Cleveland-cnss40.t3.ans.net (140.222.40.2)  23 ms  22 ms  24 ms 

     The T3 point-to-point line between the New York POP and the
     Cleveland POP is on these two T3 ports on CNSS32 and CNSS40. 

7  t3-0.Cleveland-cnss41.t3.ans.net (140.222.41.1)  22 ms  23 ms  23 ms 

     The Ann Arbor ENSS is attached to T3 Port #0 on CNSS41 (the T3
     concentrator in the Cleveland POP). 

8  t3-0.enss131.t3.ans.net (140.222.131.1)  30 ms  27 ms  27 ms 
9  merit.edu (35.1.1.42)  29 ms  39 ms  31 ms 

     The merit.edu machine is one hop from ENSS131 which is physically
     located in a University of Michigan building. 

While every effort was taken to ensure the accuracy of the DNS data,
typos and glitches may have crept in.  If you notice any problems, please
report them by sending mail to hostmaster at ans.net. 

OSI Support on T3 Backbone  
========================== 
     In early December, OSI CLNP forwarding over the T3 backbone was
configured via encapsulation of CLNP over IP packets using the EON
method (RFC1070) until native CLNP switching services are available on
the T3 routers.
     One RT from the T1 NSS node at each midlevel/peer site is being used for
the EON encapsulation function. It is our intention to keep this RT and
one other for statistics collection purposes running at each site.

CA*Net Reconfiguration Status
=============================
      CA*Net is working with Merit and ANS to complete their plan to
change their connections to the US (at Seattle, Ithaca and Princeton)
so that the US nodes will run the CA*Net RT kernel and become logically
part of the Canadian routing domain. This project has not yet been 
completed, and so currently the three NSS nodes from the T1 Backbone
(but without the T1 circuits) are being used to route traffic between
CA*Net and NSFNET. Once the software installation is complete the
circuits will be moved over to these RTs and the full NSS noes will
be disabled.

AIX 3.2 Migration Plan
====================== 
     System testing continued in December for the new AIX 3.2 operating
system for the RS/6000 routers.  This software will be installed on the
T3 test network in January for final system testing prior to deployment in
early February.

GATED Routing Software Development
==================================
     Work continued on the development of GATED as replacement for
rcp_routed software on the T3 backbone.  The same Interior Gateway
Protocol (IGP) used by rcp_routed will be supported initially on GATED as
a transitional measure to simplify the deployment of GATED.  Full IS-IS
integrated routing will follow in GATED, along with BGP4 with CIDR
routing capability.

New ANSnet Nodes Installed
==========================
ENSS       Customer        Access      Date Active
----       --------        ------      -----------
206        CERN            T1          12-01-92
187        CIX             T1          12-04-92
209        H.W. Wilson     T1          12-04-92
207        Siemens         56k         12-07-92
177        N.Y. Bank       T1          12-09-92
199        Dept. of Trans. 56k         12-11-92
179        MCNC Backup     56k         12-15-92

Cisco PROM Upgrade on ANSNET
============================ 
     We are finalizing tests that will allow us to upgrade the software
level on all ANSnet Cisco routers to 9.0(3).  This will fix several bugs that
currently exist on the current 8.3.X that is installed across the network.