Backbone Engineering Report for July

Wed Aug 12 17:59:08 UTC 1992

Hi. This was included in this month's Internet Monthly Report.
	Mark

                ANSNET/NSFNET Backbone Engineering Report

                                 July 1992

                Jordan Becker, ANS      Mark Knopper, Merit
                becker at ans.net          mak at merit.edu

T3 Backbone Status
==================
	Traffic migration from the T1 to the T3 backbone is in its
final stages, with only a few networks on T1 left to be cut over to
T3.  New software is being deployed on the T3 backbone nodes to
further improve performance and stability. Load-related problems
continue to occur on the T1 network's Routing Control Processors and
are being addressed with software changes to improve RCP efficiency.
Activities for the "Phase IV" upgrade of the T3 backbone are being
planned including FDDI adapter upgrades and other changes that will
support dismantling of the T1 backbone later this year.

Traffic Migration and Statistics
================================
	In July, 12,884,937,091 packets entered the T3 network, up 20%
from June.  4,754,950,521 packets entered the T1 network, down 17.5%
from June. The total for both networks (minus cross-network traffic)
was 17,063,064,412, up 8.4% from June.

	6130 networks are configured for the T1 backbone, and 5269 for
the T3 backbone. Actual network announcements by mid-level peer
routers to the T1 backbone are 4426 networks, and 3508 to the T3. It
is interesting to note that 1246 of the configured T1 networks were
never actually announced to the backbone during the month; and 882 of
the configured T3 networks were never announced to the backbone.  New
software has been installed to track the number of announced networks
over time.

	During July, NorthWestNet, Terrestrial Wideband (TWB) Net,
MilNet and Minnesota Supercomputer Center (MSC) Net were cut over
to the T3 backbone. The remainder of the networks for Sprint
and International Connections Manager (ICM) were cut over in early
August.

	Networks remaining to be cut over are: ESNet, Mexican
Networks (at Boulder), EASINet, and CA*Net. The first two are
expected in mid-August. The EASINet and CA*Net peers are currently
only physically connected to the T1 backbone and will be moved
to allow peering with the T3 backbone in early September.

Improvements to the T3 Backbone
===============================

New Software Build
==================
	Version 2.81.31 of the AIX software and microcode has recently
been deployed on the T3 backbone nodes. This build includes several
significant fixes to problems and improvements to performance.  These
include: a buffering change to increase performance for T3 to ethernet
interface transfers, a fix to a condition that causes an ethernet
interface to freeze, on-card support for additional ethernet MIB-II
variables, changes to the T3 card microcode, and a fix to a bug in
ICMP time-stamp processing (now resolves to the millisecond).

T3 Routing Software Changes
===========================
	Numerous improvements have been made to the T3 routing daemon,
rcp_routed.  These are described in release notes available for
anonymous ftp:

    ftp.ans.net:/pub/info/t3-rcp_routed/Release-Notes

	The most significant recent change provides a workaround for
regionals that generate a default pointing at 140.222 and find their
ENSS isolated from the rest of 140.222 T3 system due to loss of the
ENSS link to the CNSS.  Other changes include better handling of BGP
external metrics, and various reliability and operational
improvements.

	Routing stability measurements have been made over the last
few months.  This information, in addition to the detailed logs have
helped identify routing disturbances, isolate problems involving
connectivity loss (IS-IS adjacency loss or BGP disconnect) which
should not have occurred, and correct the problems.  This has resulted
in a measurable improvement in BGP stability.

FDDI Interface Upgrade Planned for August
=========================================
	We are planning to upgrade the existing FDDI interfaces
deployed on several ENSS nodes to new RS960 technology based interface
adapters. Due to the complexity of the May '92 T3 upgrade, and the
need for operational stability, it was decided not to upgrade the
older FDDI interfaces at the same time as the T3 network.  Instead, it
was decided to decouple the process of testing and deployment of the
new T3 technology from the new FDDI technology.  The FDDI interface
that is installed at each T3 ENSS location will be upgraded from an
earlier technology to the newer RS960 adapter technology in August
'92.  This will raise the peak throughput observed by end users well
beyond the current 10-14Mbps that may be observed today.  Our
objective is to continue to incrementally improve performance while
maintaining very high reliability.

	Once the new FDDI adapters are installed and operational at
the existing nodes, we will explore installations at other ENSS nodes
that do not yet support FDDI.

New SNMP Daemon Installed
=========================
	A new release of the SNMP software on the T3 backbone was
installed.  With the deployment of build 2.81.31 and SNMP version
3.213, in and out octet counts are available for the T960 ethernet and
T1 cards.  The current MIB variable map for the ethernet, T1 and T3
interfaces supported by the RS6000 nodes is as follows.

		  inU  outU  inNU  outNU  inOct outOct InErr OutErr
====================================================================
T960 Ethernet	| U+B  U+B    0      0     OK     OK   IN+OUT  0   |
RS960 T3	|  0    0     IN    OUT	   OK     OK     OK    OK  |
====================================================================

	The pending operating system upgrade of the RS/6000 routers to
AIX 3.2 will allow for more complete support of the appropriate MIBS
for each adapter type.

RS/960 Memory Parity Problem
============================
	A manufacturing problem with memory chips on a small number of
the RS960 T3 interface adapters has been identified and corrected.
However to protect against reccurrence or new introduction of memory
problems, we are scheduling diagnostics to be run on some nodes along
with other scheduled maintenance over the next few weeks.

DSU Synchronization Problem
===========================
	A problem was observed in the first few weeks after the
April-May deployment of RS/960 T3 cards and DSUs, where a T3 DSU would
lose synchronization with its peer on nodes that connect to the
Houston POP.  This problem has not been reproducible in our test labs
or in the T3 Research Network after extensive experimentation, nor has
it occurred on production nodes since then.

DSU PROM Upgrade
================
	The firmware in the T3 Technologies DSUs will be upgraded to a
new level, allowing extended remote monitoring capability.  This
upgrade will be performed in the late August timeframe with plenty of
advanced notice.

T1 ENSS Memory Upgrades
=======================
	Each of the T1 ENSS nodes on the T3 backbone were recently
upgraded from 16MB -> 32MB of main memory to provide capacity for
increased paging.

T1 Backbone Status
==================
	A new rcp_routed release was deployed early in July which
increased the BGP packet size to 4096. This was necessary to allow a
full complement of networks to be announced from the backbone to
regional peers (the same change was made to the T3 rcp_routed).

	Later in the month another new rcp_routed version was deployed
which contained changes to support compression of the IS-IS protocol
LSP. This is necessary to allow the T1 network to sustain the current
limit of 2200 networks being announced in from external mid-level
peers to a single NSS.

	NSS 11 at Houston exhibited some problems after the Milnet
networks were cut over to the T3 backbone.  Some mid-level peers
announce all of the networks learned from the T3 backbone into the T1
backbone, and these are stored by the T1 node but are marked as
unusable. This caused the Houston RCP to run out of memory. A change
was made to the peer routers to prevent these networks from being
announced to the T1 network, and a future release of the rcp_routed
will include a change so that these routes are not stored.

T3 Backbone Phase IV Planning
=============================
	Upgrades to the T3 backbone are being planned to support
future performance and functional enhancements, and to allow the
dismantling of the T1 backbone.  The major features of this plan
include:

1) T3 ENSS FDDI interface upgrades to new RS/960 card.

2) A T1 circuit will be installed at each T3 ENSS to allow
   a backup connection to a different CNSS. This will provide
   some redundancy in the case of T3 circuit or primary CNSS
   failure.

3) Network source/destination pair traffic statistics collection for
   the T3 nodes. This feature is provided on the T1 backbone
   currently.

4) Open Systems Interconnection/Connectionless Network Protocol
   (OSI/CLNP) support. This feature is provided on the T1 backbone
   currently.

5) Upgrade selected T3 ENSS ethernet interfaces to new RS/960
   ethernet card.

6) Topology Changes:
	o Moving the Washington D.C. area CNSS to a new MCI POP for
	  closer proximity to several ENSS locations.  The tail
	  circuits of the existing network attachments to this
	  POP will be reduced to local access circuits only.

	o Deploy a new CNSS in Atlanta to reduce the GA Tech T3 tail
	  to local access only, and provide expansion capability in
	  the southeast.