AS690 Gated/BGP4 Deployment

Sun Feb 13 00:10:02 UTC 1994

The good news is that since Friday morning, we now have a stable
gated/BGP4 router interoperating with 90+ rcp_routed peers within
AS690, and we are preparing for further gated/BGP4 deployment on
AS690 routers.

The bad news is that we suffered a major system-wide rcp_routed
failure along the way.  On Thursday evening around 22:00EST, we
exercised a dormant bug within rcp_routed where the gated router
(ENSS205) sent a valid IBGP message to its 90+ rcp_routed peers
that they did not process correctly and the routing daemon died. 
This resulted in about a 45 minute outage, although the gated peer
(ENSS205) stayed up.  The problem was immediately traced to an
rcp_routed bug.  The rcp_routed bug (which was not found on our
testnet) was fixed and deployed during Friday mornings configuration
window. 

The ENSS205 router is a semi-production system that has been used
to stage new gated versions onto AS690.  The last update to gated
on this system was done outside the normal configuration
maintenance window in an effort to expedite the deployment on other
full-production nodes for CIDR transition.  While this outage could
have occurred at any time (it was induced by a random external peer
interaction) a procedural improvement will be administered for all
gated installations by establishing special scheduled maintenance
windows to minimize the impact of any problems, and to work
towards a timely AS690 CIDR deployment.

There are a two non-critical known problems with the AS690 gated
that have been (or will be addressed) prior to the next installation. 
I have summarized these problems below for those interested.

Another new development is that the rcp_routed support for a default
route that was previously deployed on AS690 was successfully tested
during the last scheduled configuration maintenance window.  A non-
disruptive configuration change can be administered at any time to
enable AS690 to default to the AS1133 (gated BGP4/CIDR) peer
which is connected to both MAE-East, and to CNSS57 (AS690) over
a private 10Mb/s ethernet segment.  This may be required to assist
specific MAE-East peers with a timely CIDR deployment.

We have established two specific upcoming maintenance windows for
gated deployment on AS690 routers.  The next window will be
Sunday Morning Feb 14 (00:30EST - 08:00EST).  The AS690 routers
that we have picked as candidates to convert to run gated (pending
consent of the peers) include: 

ENSS194 (ANS Ann Arbor Backup Router)
ENSS158 (Maui HPCC) 
CNSS120 (Honolulu CNSS)  
ENSS205 (new Gated re-deployment) 
ENSS160 (ANS Elmsford) 
ENSS139 (Rice U) 
ENSS131 (Ann Arbor) 

There is an overlapping power maintenance scheduled window for the
MCI New York POP (Sunday morning 00:30EST-02:30EST) that we
will work around, and should not interfere with the gated deployment
activities.

The ANS NOC has contacted each of the peers that will be affected
by service disruptions on these nodes to acknowledge their consent,
and NSR messages have been sent for each of these maintenance
windows describing the gated deployment, and the potential for
AS690 routing instabilities that may occur during these windows. 

Following this maintenance we would like to validate with the peers
that all configurations with external peers are behaving as expected. 
If this maintenance is successful we would like to continue the AS690
deployment during the Tuesday morning Feb 15th configuration
maintenance window (05:00EST - 08:00EST) for the following
candidate nodes: 

ENSS136 (College Park) 
ENSS145 (FIX-East) 
ENSS144 (FIX-West) 

We would try to identify any problems within the maintenance
window.  If we could not complete this maintenance within the
window, these routers would be rolled back to run rcp_routed. 

Once we have stabilized the above set of nodes, the rest of the
AS690 could proceed as rapidly as can be scheduled (within
maintenance windows).

For general interest, the two known problems with AS690 gated that
we expect to fix before the AS690 deployment concludes.  These
may be summarized as:

1.   Jurassic LSPs.  At the time rcp_routed was originally developed,
     there was an IS-IS LSP packet format that was designed to
     carry external routes, before IBGP was implemented to carry
     these.  Modern rcp_routed does no longer cares about this,
     however still occasionally generates these "Jurassic LSPs" which
     have header but carry no external routes, from specific
     production rcp_routed nodes at rcp_routed startup time (we can
     not reproduce this on our testnet). Gated does not know what
     to do with these packets so it drops them and logs an error
     message.  Rcp_routed acks them and goes on about its
     business.  Rather than modifying rcp_routed to eliminate these
     LSPs, we instead worked around this by adding a few lines of
     code to the gated SLSP to ack these packets and ignore them. 
     This does not cause problems running gated on a few
     production network nodes and will not bother ENSS205, but we
     should and will fix this before we scale the deployment up
     further.

2.   BGP memory leak.  We have previously known from our testnet
     experience that there is a slow memory leak in the gated BGP
     code that is exercised when a connection to an external peer
     fails and gated tries to re-establish the session.  The only way
     we could observe this problems on the testnet was when a
     gated machine was forced to try and establish sessions (2 times
     per second) with 80 external peers that refused to connect for
     about 4 days.

     We would not expect to see problems with this on nodes that
     are configured properly, but in any case we expect to have a
     fix for this shortly and this will be retrofitted on existing gated
     nodes during a future installation window.