Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

Tue May 2 14:54:32 UTC 2023

We have upgraded NX-OS to a new major version and have the same results. 

Apr 20 09:36:05 UTC: %UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 consistency checker FAILED on slot 1 
Apr 19 13:55:57 UTC: %IPFIB-2-FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for IPV4 routes in LPM table 

show forwarding inconsistency 
... 
69. slot(1), vrf(default), prefix(198.XXX.XXX.0/YY), Route inconsistent in FIB Software. 

Well, those seem like problems. 

EASTGATE401-BGP-02# show ip route summary 
IP Route Table for VRF "default" 
Total number of routes: 292 
Total number of paths: 292 

Unicast paths: 
Best paths per protocol: Backup paths per protocol: 
am : 48 None 
local : 8 
direct : 8 
static : 5 
broadcast : 19 
bgp-XXXXX : 204 

Number of routes per mask-length: 
/0 : 1 /8 : 1 /18: 2 /19: 4 /20: 26 
/21: 24 /22: 35 /23: 18 /24: 100 /27: 1 
/28: 1 /29: 1 /30: 4 /32: 74 

It's choking on only 292 routes? 

Error Message FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for IPV4 routes in LPM table 
Explanation The TCAM device in the Layer 3 forwarding ASIC has reached its system limits for IPv4 entries in the LPM table. 

Recommended Action No action is required. 

TCAM is exhausted and no action is recommended? 

Error Message UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 consistency checker FAILED on slot [chars] 
Explanation FIB Ipv4 route consistency checker Failed. Route database is consistent with hardware 

Recommended Action No action is required. 

The FIB is inconsistent and no action is recommended? 

EASTGATE401-BGP-02# show system internal forwarding route summary 

slot 1 
======= 

IPv4 hosts & routes summary on module 1 
--------------------------------------------- 

Max host route entries : 8192 
Total number of IPv4 host routes used: 72 
Max LPM table entries : 7167 
Total number of IPv4 LPM routes used : 16 

I seem to be out of my depth here in that 292 is less than 7167, but yet it still fails. 

----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

From: "Mike Hammett" <nanog at ics-il.net> 
To: "NANOG" <nanog at nanog.org> 
Sent: Monday, April 3, 2023 12:21:29 AM 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. 

I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and it has the expected interface. 

However, sFlows show the packets leaving on a different interface, the one that would carry the default route for routes not otherwise known. 

If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected interface? 

----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20230502/d8785bac/attachment.html>