Followup: anyone else seeing very long AS paths?

Fri Feb 20 20:55:35 UTC 2009

I know this is getting boring, but I don't want to see information lying
around hinting that the ISPs around the world were to blame for the Monday
incident due to their sloppy software upgrade policies. That's not the case;
a lot of very recent IOS releases were affected.

There was lots of conflicting information flowing around. Rodney did an
excellent job with the bug description, but I was still wondering what
exactly was going on, so I've tested all potential combinations of
inbound/outbound IBGP/EBGP prepending/no-prepending cases to identify
exactly what can happen; you should be able to figure out which one affected
your network:

http://blog.ioshints.info/2009/02/oversized-as-paths-cisco-ios-bug.html

And it should be reiterated that anyone that has already configured bgp
maxas-limit has avoided this problem.

Ivan

> -----Original Message-----
> From: Rodney Dunn [mailto:rodunn at cisco.com] 
> Sent: Thursday, February 19, 2009 9:15 PM
> To: Rodney Dunn
> Cc: German Martinez; Ivan Pepelnjak; nanog at nanog.org
> Subject: Re: anyone else seeing very long AS paths?
> 
> We are working on a document for Cisco.com but in the interim 
> here is the bug that will fix the issue of a Cisco IOS device 
> sending an incorrectly formatted BGP update when as a result 
> of prepending it goes over 255 AS hops.
> 
> Note: The Title and Release-note on bug toolkit may be a bit 
> different as I just updated it to be more accurate.
> 
> Of all the scenarios I've looked at (thanks to those that responded
> offline) there wasn't a condition found where this could 
> happen without AS path prepending being used.
> 
> Please respond offline or let's move the discussion over to 
> cisco-nsp at this point.
> 
> CSCsx73770
> Invalid BGP formatted update causes peer reset with AS prepending
> 
> <B>Symptom:</B>
>  
>  A Cisco IOS device that receives a BGP update message and as 
> a result of AS prepending needs to send an update downstream 
> that would have over 255 AS hops will send an invalid 
> formatted update. This update when received by a downstream 
> BGP speaker triggers a NOTIFICATION back to the sender which 
> results in the BGP session being reset.
>  
>  <B>Conditions:</B>
>  
>  This problem is seen when a Cisco IOS device receives a BGP 
> update and  due to a combination of either inbound, outbound, 
> or both AS prepending it needs to send an update downstream 
> that has more than 255 AS hops.
>  
>  <B>Workaround:</B>
>  
>  The workaround is to implement <CmdBold> bgp maxas-limit X 
> <NoCmdBold> on the device that after prepending would need to 
> send an update with over 255 AS hops. Since IOS limits the 
> inbound prepending value to 10 the most that could be added 
> iss 11 AS hops (10 on ingress, 10 on egress, and 1 for normal 
> eBGP AS hop addition). Therefore, a conservative value to 
> configure would be 200 to prevent this condition.
>  
>  
> 
> Full support of Section 5.1.2 of RFC4271 is being tracked under
> CSCsx75937
> Add BGP support of AS paths longer than 255 per Section 5.1.2 
> of RFC4271
> 
> Thanks to those that worked offline with us to verify the 
> field results reported.
> 
> Rodney
> 
> 
> 
> 
> On Tue, Feb 17, 2009 at 05:27:01PM -0500, Rodney Dunn wrote:
> > If you want to take this offline send it unicast or we 
> could move it 
> > to cisco-nsp.
> > 
> > What scenarios are you seeing that appear broken other than when a 
> > notification is sent when a > 255 hop update is received?
> > That's the one I'm working on right now.
> > 
> > Rodney
> > 
> > On Tue, Feb 17, 2009 at 05:31:49PM -0500, German Martinez wrote:
> > > On Tue Feb 17, 2009, Rodney Dunn wrote:
> > > 
> > > Hello Rodney,
> > > It will be great if you can share with us your findings.  
> It seems 
> > > like we are hitting different bugs in different platforms.
> > > 
> > > Thanks
> > > German
> > > 
> > > > Ivan,
> > > > 
> > > > It is confusing but from what I have tested you have it correct.
> > > > 
> > > > The confusing part comes from multiple issues.
> > > > 
> > > > a) The documentation about the default maxas limit 
> being 75 appears to be
> > > >    incorrect. I'll get that fixed.
> > > > 
> > > > b) Prior to CSCee30718 there was a hard limit of 255. 
> After that fix
> > > >    AS sets of more than 255 should work.
> > > > 
> > > > c) CSCeh13489 implemented the maxas command to mark it 
> as invalid and
> > > >    not send.
> > > > 
> > > > 
> > > > There does appear to be an issue when you cross the 255 
> boundary 
> > > > and the next hop router sends a notification back.
> > > > 
> > > > I've got it recreated in the lab and we are working to clearly 
> > > > understand why that is. I'll post an update once we have more.
> > > > 
> > > > The way to prevent it is the upstream device that 
> crosses the 255 
> > > > boundary on sending needs to use the maxas limit 
> command to keep it less than 255.
> > > > 
> > > > It doesn't work on the device that receives the update 
> with the AS 
> > > > path larger than 255.
> > > > 
> > > > Rodney
> > > > 
> > > > On Tue, Feb 17, 2009 at 08:58:48PM +0100, Ivan Pepelnjak wrote:
> > > > > > We were dropping ALL prefixes and the eBGP session 
> was still 
> > > > > > resetting.
> > > > > 
> > > > > Upstream or downstream?
> > > > > 
> > > > > > 1) "bgp maxas-limit 75" had no effect mitigating 
> this problem 
> > > > > > on the IOS we were using. That is: it was 
> previously verified 
> > > > > > to be working just fine to drop paths longer than 
> 75, but once 
> > > > > > we started receiving paths >
> > > > > > 255 then BGP started resetting.
> > > > > 
> > > > > I was able to receive BGP paths longer than 255 on 
> IOS release 
> > > > > 12.2SRC. The paths were generated by Quagga BGP daemon.
> > > > > 
> > > > > 12.2SRC causes the downstream session to break when the 
> > > > > installed AS-path length is close to 255 and you use 
> downstream AS-path prepending.
> > > > > 
> > > > > In your case, I'm assuming you were hit with an older bug 
> > > > > (probably at the
> > > > > 128 AS-path length boundary). It would be very hard 
> to generate 
> > > > > just the right AS-path length to unintentionally break your 
> > > > > upstream EBGP session (as I said before, it's a nice targeted 
> > > > > attack if you know your downstream topology).
> > > > > 
> > > > > If your IOS is vulnerable to the older bugs that 
> break inbound 
> > > > > processing of AS paths longer than 128, there's 
> nothing you can 
> > > > > do on your end. The internal BGP checks reject the inbound 
> > > > > update before the inbound filters (or bgp 
> maxas-limit) can touch it and reset the upstream BGP session.
> > > > > 
> > > > > Hope this helps
> > > > > Ivan
> > > > > 
> > > > > Disclaimer: as I don't have internal access to Cisco, 
> all of the 
> > > > > above is a result of lab tests.
> > > > > 
> > 
> 