<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><br><br><div dir="ltr">Sent from my iPhone</div><div dir="ltr"><br><blockquote type="cite">On Apr 16, 2023, at 10:05, Mike Hammett <nanog@ics-il.net> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000"><span style="font-size: 13.3333px;">We did have our common upstream provider perform maintenance that then afterwards, had the traffic flowing on the right path. Later activity on our direct connection pushed it back to the common upstream. We haven't yet had the opportunity to bump our BGP session with the common upstream provider, but I suspect that will put the traffic back onto the right path. Seems like the router is just hanging onto the oldest BGP session it has, regardless of any other parameter or configuration.</span><div style="font-size: 13.3333px;"><br></div><div style="font-size: 13.3333px;">This seems like a bug. We do intend on upgrading NX-OS, but that's on someone else's schedule.</div><br></div></div></blockquote>Not that familiar with nexus 3k but I would compare the route in the rib and that on the module. If the platform is exhausting unicast route entries the control plan may show the route when module / asic doesn’t have it installed. It’s not always obviously when these things run out of tcam <div><br></div><div><a href="https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/9-x/unicast/configuration/guide/l3_cli_nxos/l3_manage-routes.pdf">https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/9-x/unicast/configuration/guide/l3_cli_nxos/l3_manage-routes.pdf</a></div><div><br></div><div>This is nexus 9k </div><div> </div><div>But I would expect it to be broadly similar in terms of diagnostic on the other platforms</div><div><br></div><div>Nexus 3ks are Broadcom merchant silicon (trident/tomahawk so I’d expect the them to run out of fib in the mid tens of thousands of routes..</div><div><br><blockquote type="cite"><div dir="ltr"><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000"><div><span name="x"></span><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline !important;float:none">-----</span><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline !important;float:none">Mike Hammett</span><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.ics-il.com/" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer">Intelligent Computing Solutions</a><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/ICSIL" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/fbicon.png" style="border:0pt none" data-unique-identifier=""></a><a href="https://plus.google.com/+IntelligentComputingSolutionsDeKalb" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/googleicon.png" style="border:0pt none" data-unique-identifier=""></a><a href="https://www.linkedin.com/company/intelligent-computing-solutions" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/linkedinicon.png" style="border:0pt none" data-unique-identifier=""></a><a href="https://twitter.com/ICSIL" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/twittericon.png" style="border:0pt none" data-unique-identifier=""></a><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.midwest-ix.com/" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer">Midwest Internet Exchange</a><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/mdwestix" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/fbicon.png" style="border:0pt none" data-unique-identifier=""></a><a href="https://www.linkedin.com/company/midwest-internet-exchange" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/linkedinicon.png" style="border:0pt none" data-unique-identifier=""></a><a href="https://twitter.com/mdwestix" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/twittericon.png" style="border:0pt none" data-unique-identifier=""></a><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.thebrotherswisp.com/" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer">The Brothers WISP</a><br style="color:rgb( 0 , 0 , 0 );font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/thebrotherswisp" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/fbicon.png" style="border:0pt none" data-unique-identifier=""></a><a href="https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg" style="font-family:'times new roman';font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" rel="nofollow noopener noreferrer"><img src="http://www.ics-il.com/images/youtubeicon.png" style="border:0pt none" data-unique-identifier=""></a><span name="x"></span><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Mike Hammett" <nanog@ics-il.net><br><b>To: </b>"NANOG" <nanog@nanog.org><br><b>Sent: </b>Monday, April 3, 2023 12:21:29 AM<br><b>Subject: </b>Cisco Nexus 3k Route Selection\Packet Forwarding Debugging<br><br><style>p { margin: 0px; }</style><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000"><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);">We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that.</span><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);"> </span><br style="font-family: monospace; font-size: medium;"><br style="font-family: monospace; font-size: medium;"><br style="font-family: monospace; font-size: medium;"><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);">I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface.</span><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);"> </span><br style="font-family: monospace; font-size: medium;"><br style="font-family: monospace; font-size: medium;"><br style="font-family: monospace; font-size: medium;"><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);">However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known.</span><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);"> </span><br style="font-family: monospace; font-size: medium;"><br style="font-family: monospace; font-size: medium;"><br style="font-family: monospace; font-size: medium;"><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);">If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface?</span><span style="font-family: monospace; font-size: medium; background-color: rgb(255, 255, 255);"> </span><br><br><div><span></span><br><br>-----<br>Mike Hammett<br>Intelligent Computing Solutions<br>http://www.ics-il.com<br><br>Midwest-IX<br>http://www.midwest-ix.com<span></span><br></div><br></div></div><br></div></div></blockquote></div></body></html>