Global Crossing Contact / BGP and SONET interaction question

Wed Jul 26 04:55:20 UTC 2006

Two somewhat intertwined questions.  I'll ask the second part first.

I buy transit from Global Crossing and another carrier on HDLC 
encapsulated DS3's.

Recently my BGP session has started flapping on the GX circuit... It 
looks something like this:

Jul 21 21:17:43.731 UTC: %BGP-3-NOTIFICATION: received from neighbor 
67.17.168.73 6/6 (cease) 0 bytes
Jul 21 21:17:43.731 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down 
BGP Notification received
Jul 21 21:18:25.439 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up
Jul 21 21:29:52.315 UTC: %BGP-3-NOTIFICATION: received from neighbor 
67.17.168.73 6/6 (cease) 0 bytes
Jul 21 21:29:52.315 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down 
BGP Notification received
Jul 21 21:30:38.511 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up
Jul 21 21:31:34.411 UTC: %BGP-3-NOTIFICATION: received from neighbor 
67.17.168.73 6/6 (cease) 0 bytes
Jul 21 21:31:34.411 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down 
BGP Notification received
Jul 21 21:32:20.535 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up
Jul 21 21:32:52.547 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Down 
Peer closed the session
Jul 21 21:33:32.703 UTC: %BGP-5-ADJCHANGE: neighbor 67.17.168.73 Up

There are no other log entries during the periods when this occur. 
Unfortunately this causes enough prefix flaps that any prefixes which 
are preferred through GX are damped for like a half hour by certain 
providers as my BGP routes get added/withdrawn through the GX link.

GX claims (although I'm not sure they really know) that these are caused 
by SONET ring switches.  I can believe this, since I haven't seen any 
real circuit flaps, and my understanding is that a SONET switch should 
generally be fast enough that you normally won't see the transition 
other than perhaps an error counter or two cranking up.  However, it 
seems strange that I'm getting a 6/6 (cease) notification which I read 
as "configuration change" from their router.   GX also seems to be at a 
loss to explain why my BGP is flapping - other than to point at the 
SONET switches.

I guess I'm trying to find out if someone on the list recognizes what 
this might be so I can perhaps help GX find and fix this.   I'm also 
kinda curious as to whether or not typically a SONET ring switch event 
would actually propagate into a router in such a way that BGP would try 
to shut down the BGP sessions.   I'm just having a hard time visualizing 
how a supposedly below-layer-two switch would cause bgp to reset in this 
manner.  Not being a SONET expert even by any long stretch of the 
imagination leaves me with some holes here, but I thought the whole goal 
of SONET when used to provide DS3 circuits was to hide the ring switches 
as much as possible from the DS3 circuits - realizing that framing may 
be hard to preserve on a ring switch which would cause momentary loss of 
sync or similar - which usually shows up as an error instead of a 
interface flap.

And finally, does anyone have a contact within GX with a clue?  So far 
I'm not sure I've talked to anyone who knows anything but how to spell 
BGP.  I'd really like to talk to someone about the real cause of these 
flaps and try to resolve them so they don't reoccur.

-forrest