May ANSNET/NSFNET Backbone Engineering Report

Tue Jun 9 04:09:34 UTC 1992

Hi. Following is an excerpt from the May report which will be
included in this month's Internet Monthly Report. The missing
text was already sent to this list as RS/960 deployment notes
each week during the backbone hardware upgrades.
	Mark

+----------------------------------------------------------------+


                ANSNET/NSFNET Backbone Engineering Report

                                 May 1992


                Jordan Becker, ANS      Mark Knopper, Merit
                becker at ans.net          mak at merit.edu


T3 Network Status
=================


	The major event for May was the successful completion of
the deployment of new "RS/960" DS3 interface cards on the entire
T3 backbone. The five week deployment went very smoothly, and 
the new cards have increased the reliability and performance of
the network. Cutovers of traffic from the T1 to the T3 backbones were
suspended and delayed until June because of the deployment activities.
Other activities of interest include new build and routing daemon
software on the T3 network nodes.

	Two of the three links to CA*Net (Toronto to Ithaca,
and Montreal to Princeton) were upgraded from fractional-T1
rate to full T1 rate this month.

	The T1 backbone is stable now that it is carrying less
than 50% of the total NSFNET traffic, though the RCP nodes continue
to be closely monitored since they are still carrying the full
routing advertisements. 

	Full traffic statistics for the T3 net are not available
because the initial software release for the RS/960 support on
the T3 backbone did not support returning the interface counters from
the SNMP MIB. This will be corrected for the month of June. Also,
the net to net matrix statistics are not available. A sampling 
technique is being developed which will allow this data to be
collected once again. 

	
Routing Daemon Changes
======================

	The rcp_routed program for the T3 network was upgraded
to a new version containing several improvements. These included
a change to allow Cisco peer routers to accept a full routing
table (~4000 networks) from the ENSS. There is now an option 
for rcp_routed to add a time delay when the peer is a slower
processor. All Ciscos in the backbone now have a full routing
table rather than using default. Another change was made to
fix a problem that caused an internal-BGP connection to be reset.
Several bug fixes that causes crashes or daemon restarts were
fixed.


SNMP Daemon Changes
===================

	The RS/960 software build does not include the ability to
obtain ethernet interface statistics. This will be corrected the
week of June 8. As of June 3, traffic statistics were available
for the ENSS T3 interfaces. These figures can be used to closely
approximate traffic counts for the ENSS nodes. It should be noted
that both the if[In|Out]McastPkts and if[In|Out]UcastPkts variables
should be queried and summed for both T3 and ethernet interfaces,
for those who are polling the backbone nodes.


RS/960 Deployment Chronology
============================

	Following is a summary of the 5 week deployment activities
for May which completed installation of the RS/960 DS3 hardware
adaptors in all T3 network nodes.


...


Step 5: May 22-23


	The fifth, and final, stage of the Phase 3 deployment was commenced at
23:00 hours Friday, May 22 and completed by turning on all traffic through a
re-built, RS960 based backbone at 09:38 Saturday morning. The problems
encountered were relatively minor and were dealt with rather quickly, enabling
return to full operation ahead of schedule.

	The resulting stability of the T3 network has been excellent since
this final cutover.  We are very grateful to all of the regional-techs that
assisted and cooperated with us during the 5 week deployment.


Week 5 Problems
===============
	At Washington POP (CNSS59) after the rebuild, the machine failed to
reboot properly in "SECURE" mode. The service switch was found to be defective
and was replaced. CNSS57, targeted to be a spare, was cannibalized for the
spare part.

	At Washington POP (CNSS58) the RS960 T3 card in slot 4 failed to load
its microcode and the card was replaced.

	At Washington POP (CNSS56) the node did not recognize an RS960 T3 card
(no POSID).  This indicates that card was not identifying itself to the system
and the card was replaced.

	At College Park (ENSS136) the RS960 T3 card came up in NONAP mode
(would not talk to the system) and was replaced.

	At Greensboro POP (CNSS73) a corrupted ODM database was suspected and
the node system software was reloaded from a prepared tape. Deborah
Matties-Sharp and Kraig Owen, in anticipation, had all sites prepared with
bootable back-up tapes and their foresight paid off.  Also at Greensboro, the
communication cable from the DSU servicing Georgia Tech. (ENSS138) was found
to be defective (this cable previously was operational) and was replaced by
removing an identical cable from CNSS75 which no longer needed a DSU.  The
problem was difficult to isolate because the DSU would respond to query of the
serial number, but would respond with garbaled messages to other queries.

	It was noted that at Greensboro LBO's had been set at "LONG" (or HI)
but after the DSU change the T3 circuit to Georgia Tech had to be set "SHORT"
(or LOW).  We are investigating this further.

	Lastly, at New York POP the cable from the RS6000 serial port#S2 for
the CNSS35 modem was replaced as planned.


Results
=======
	It took 27 minutes to have all routes come up from the first site
(Hartford POP) to the last site Georgia Tech. (ENSS138).

	We have installed new SNMP agent (and sub-agent) software across the
T3 backbone to support the new variables collected on the RS960 T3 cards

	SNMP access for some variables on the T960 ethernet/T1 cards are
temporarily in-accessible.  We expect to have this supported shortly.  SNMP
client programs may have to be re-tested to insure they are collecting valid
data.

	Because there are no more hybrid links (e.g. Hawthorne T3<->RS960 T3),
the T3 network link metrics have been reset to normal values in order to split
traffic equally across equal hop paths.

	A party was held at Mark Knopper's house on May 23 to commemorate
the complete deployment of the RS/960 cards. As part of the festivities,
an authentic Hawthorne card was buried amid a not-so-solemn ceremony.