<Keepalives are temporarily in throttle due to closed TCP window>

Wed Sep 16 23:47:10 UTC 2009

>Either a) you have the mtu misconfigured on that 7206vxr

That part is where I am at a loss.  How is it the 6509 can establish a
IBGP session with a 7606 when it has to go through the 7206 VXR?  The
DS-3s are connected to the 7206 VXR. To add more depth to the story.  I
have 8 IBGP sessions that are connected to the 7206 VXR that have been
up and running for over a year.  Some of the sessions traverse the DS-3s
and or a GigE long haul connections.  There are a total 10 Core routers
that are mixture of Cisco 7606, 6509s, 7206 VXR w/ NPE400s or G1s.  Only
this one IBGP session out of 9 routers is not being established.  Since
I have a switch between the 7606 and 7206, I plan to put a packet
capture server and see what I can see. 

-----Original Message-----
From: Richard A Steenbergen [mailto:ras at e-gerbil.net] 
Sent: Wednesday, September 16, 2009 2:07 PM
To: Michael Ruiz
Cc: Brian Dickson; nanog at nanog.org
Subject: Re: <Keepalives are temporarily in throttle due to closed TCP
window>

On Wed, Sep 16, 2009 at 01:18:20PM -0500, Michael Ruiz wrote:
> Here is the network architecture from the Cisco 6509 to the 7206 VXR.
> The 6509 has a successful BGP session established with another router,
> Cisco 7606 w/ Sup720-3bxls.  The 7606 and 7206 VXR are connected
> together by a Cisco 3550 switch. In order for the 6509 to establish
the
> IBGP session to the 7606, it has to pass through two DS-3s, go through
> the 7206 VXR, out the Fast E, through the Cisco 3550, and then to the
> 7606. I checked the MTUs on the 3550s and I am seeing the Fast E
> interfaces are still showing 1500 bytes. Would increasing the MTU size
> on the switches cause any harm? 

As other people have said, this definitely sounds like an MTU problem. 
Basically you're trying to pass 4470 byte BGP packets over a link that
drops anything bigger than 1500. The session will establish because all
the setup packets are small, but the tcp session will stall as soon as
you try to send routes across it.

What should be happening here is the 6509 will generate a 4470 byte
packet because it sees the directly connected interface as a DS3 and
doesn't know the path is incapable of supporting > 1500 bytes end to
end. The layer 3 device on the mtu choke point, in this case the faste
interface on the 7206vxr, should be configured to a 1500 byte mtu. This
will cause the 7206vxr to generate an ICMP neegfrag when the 4470 byte
packet comes along, and cause path mtu discovery to lower the MSS on the
IBGP session. Either a) you have the mtu misconfigured on that 7206vxr
port, b) your router is misconfigured not to generate the icmp, c)
something in the middle is misconfigured to filter this necessary icmp
packet, or d) some other screwup probably related to one of the above.

Generally speaking increasing the MTU size on a switch can never hurt
anything, but having an insufficiently large MTU on the switch is what
will break you the most (as is happening here). The problem occurs when
you increase the MTU on the layer 3 routers to something beyond what the
layer 2 link in the middle is capable of supporting. Layer 3 devices
will either fragment (deprecated) or generate ICMP NeedFrags which will
cause path MTU discovery to shrink the MSS. Layer 2 devices are
incapable of doing this, so you MUST NOT set the layer 3 MTU above what
the layer 2 link is capable of handling.

Now that said, increasing the mtu on the 3550 won't work here because
3550 MTU support is terrible. The only option you have is to configure
the MTU of all interfaces to 1546 with the "system mtu 1546" command,
followed by a reload. This is not big enough to pass your 4470 byte
packets, and will also break any MTU dependent configuration you might
be running. For example, after you do this, any OSPF speakers on your
3550 will have to have their MTUs adjusted as well, or OSPF will not
come back up due to the interface mismatch. For more details see:

http://www.cisco.com/en/US/products/hw/switches/ps700/products_configura
tion_example09186a008010edab.shtml#c4

Your best bet (in order of most preferable to least) is to a) fix
whatever is breaking path mtu discovery on the 7206vxr in the first
place, b) force the mss of the ibgp session to something under 1460, or
c) lower the mtu on the ds3 interface to 1500.

-- 
Richard A Steenbergen <ras at e-gerbil.net>
http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1
2CBC)