Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

Tue Apr 4 22:01:12 UTC 2023

Not that it's a "Fix" but have you tried rebooting the box? If this is a
bug in the forwarding plane that might clear/rebuild it. And maybe it works
correctly after that.

Friend saw something similar on a Juniper MX with DPC cards that had run
out of FIB space. It would show correctly in all places, but was failing to
install in FIB (Router cut a error about it in the log). So the actual
traffic didn't follow the same path. I saw you specifically referenced
"forwarding" in one of your copy pastes. And it looked right. But I don't
know enough about Cisco to say if that's really what is in FIB. Or just
what it thinks is in FIB.

On Tue, Apr 4, 2023 at 5:47 PM Mike Hammett <nanog at ics-il.net> wrote:

> Via all mechanisms I could find in the router, it thinks the best path is
> the direct path, the packets just don't go that way.
>
> The in traffic isn't a concern at this time, just the out (from my
> perspective).
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> ------------------------------
> *From: *"David Bass" <davidbass570 at gmail.com>
> *To: *"Mike Hammett" <nanog at ics-il.net>
> *Cc: *"Matthew Huff" <mhuff at ox.com>, "NANOG" <nanog at nanog.org>
> *Sent: *Tuesday, April 4, 2023 11:39:26 AM
> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> If you are both connected to the same upstream, but the customer wants
> traffic destined to the upstream to go through you (in and out), then they
> need to do something on their devices to try and affect the inbound path to
> their AS. From the upstream carrier in question they’ll take the best path
> to a prefix, which direct connection is generally going to be preferred
> over a transit AS (basic BGP best path algorithm stuff) unless there is
> some manipulation of the prefix advertisement happening.
>
> To confirm the path being taken you should be able to do a few trace
> routes from various locations as well as use looking glasses.
>
> Now the sflow data is an entirely different thing to analyze.
>
> On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett <nanog at ics-il.net> wrote:
>
>>
>> sh ip bgp neighbor advertised-routes shows the only routes being
>> advertised to Y are the routes that should be advertised to them. I checked
>> a variety of other peers and have the expected results.
>>
>>
>>
>> From my perspective:
>>
>> Packets come in on port A, supposed to leave on port X, but they leave on
>> port Y. All of the troubleshooting steps I've done on my own (or suggested
>> by mailing lists) say the packets should be leaving on the desired port X.
>>
>>
>> From the customer's perspective, they're supposed to be coming from me on
>> port X, but they're arriving on port Y, another network.
>>
>> Port X in both scenarios is our direct connection, while port Y is a
>> mutual upstream provider.
>>
>>
>> Without knowing more about the specific platform, it seems to me like a
>> bug in the platform. If all indicators (not just configurations, but show
>> commands as well) say the packet should be leaving on X and it leaves on Y,
>> then I'm not sure what else it could be, besides a bug.
>>
>>
>>
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>> ------------------------------
>> *From: *"David Bass" <davidbass570 at gmail.com>
>> *To: *"Mike Hammett" <nanog at ics-il.net>
>> *Cc: *"Matthew Huff" <mhuff at ox.com>, "NANOG" <nanog at nanog.org>
>> *Sent: *Monday, April 3, 2023 9:12:52 PM
>>
>> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>
>> You said that they are seeing traffic from another upstream…are you
>> advertising the prefix to them?  Are you advertising their prefix to your
>> upstream?
>>
>> Looks like the route maps are involved in some dual redistribution…might
>> want to make sure everything is matching correctly, and being advertised
>> like you want.
>>
>> On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett <nanog at ics-il.net> wrote:
>>
>>> I don't see any route-maps applied to interfaces, so there must not be
>>> any PBR going on. I only see ACLs, setting communities, setting local pref,
>>> etc. in the route maps that are applied to neighbors.
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>> ------------------------------
>>> *From: *"Mike Hammett" <nanog at ics-il.net>
>>> *To: *"Matthew Huff" <mhuff at ox.com>
>>> *Cc: *"NANOG" <nanog at nanog.org>
>>> *Sent: *Monday, April 3, 2023 8:26:30 AM
>>>
>>> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding
>>> Debugging
>>>
>>> Only two VRFs, default and manangement. IIRC, everything I saw before
>>> mentioned the default VRF.
>>>
>>> I do see a ton of route-maps. It's mostly Greek to me, so I'll have to
>>> dig through this a bit to see what's going on.
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>> ------------------------------
>>> *From: *"Matthew Huff" <mhuff at ox.com>
>>> *To: *"Mike Hammett" <nanog at ics-il.net>
>>> *Cc: *"NANOG" <nanog at nanog.org>
>>> *Sent: *Monday, April 3, 2023 8:06:51 AM
>>> *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding
>>> Debugging
>>>
>>> What about VRFs and/or policy based routing?
>>>
>>> switch-core1# show vrf
>>> VRF-Name                           VRF-ID State   Reason
>>>
>>> default                                 1 Up      --
>>>
>>> management                              2 Up      --
>>>
>>>
>>> switch-core1# show route-map
>>> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10
>>>   Match clauses:
>>>     interface: Ethernet1/33
>>>     route-type: internal
>>>   Set clauses:
>>>     metric 40000000 10 255 1 1500
>>> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20
>>>   Match clauses:
>>>     interface: Ethernet1/34
>>>     route-type: internal
>>>   Set clauses:
>>>     metric 40000000 30 255 1 1500
>>> route-map rmap_static_to_eigrp, permit, sequence 10
>>>   Match clauses:
>>>     ip address prefix-lists: prefix_static_to_eigrp
>>>   Set clauses:
>>> route-map rmap_static_to_eigrp_v6, permit, sequence 10
>>>   Match clauses:
>>>     ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp
>>>   Set clauses:
>>>
>>>
>>>
>>> From: Mike Hammett <nanog at ics-il.net>
>>> Sent: Monday, April 3, 2023 9:00 AM
>>> To: Matthew Huff <mhuff at ox.com>
>>> Cc: NANOG <nanog at nanog.org>
>>> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>>
>>> It could be an sFlow bug, but I come at this from a reported problem and
>>> gathering data on that problem as opposed to looking at data for problems.
>>>
>>> The snmp if index reported by the Nexus matches the if index in
>>> ElastiFlow.
>>>
>>>
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>> ________________________________________
>>> From: "Matthew Huff" <mailto:mhuff at ox.com>
>>> To: "Mike Hammett" <mailto:nanog at ics-il.net>
>>> Cc: "NANOG" <mailto:nanog at nanog.org>
>>> Sent: Monday, April 3, 2023 7:50:08 AM
>>> Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>> SFlow misconfiguration or bug on either the nexus or the sflow monitor?
>>> On the monitor, can you verify that the snmp interfaces are mapped to the
>>> correct ones on the nexus?
>>>
>>>
>>>
>>>
>>>
>>> From: Mike Hammett <mailto:nanog at ics-il.net>
>>> Sent: Monday, April 3, 2023 8:47 AM
>>> To: Matthew Huff <mailto:mhuff at ox.com>
>>> Cc: NANOG <mailto:nanog at nanog.org>
>>> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>>
>>> It shows the desired result.
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>> ________________________________________
>>> From: "Matthew Huff" <mailto:mhuff at ox.com>
>>> To: "Mike Hammett" <mailto:nanog at ics-il.net>, "NANOG" <mailto:
>>> nanog at nanog.org>
>>> Sent: Monday, April 3, 2023 5:38:23 AM
>>> Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>>
>>> switch-core1# sh forwarding route x.x.x.x
>>>
>>> slot  1
>>> =======
>>>
>>>
>>> IPv4 routes for table default/base
>>>
>>>
>>> ------------------+-----------------------------------------+----------------------+-----------------+-----------------
>>> Prefix            | Next-hop                                | Interface
>>>            | Labels          | Partial Install
>>>
>>> ------------------+-----------------------------------------+----------------------+-----------------+-----------------
>>> x.x.x.x/24      x.x.x.250                            Ethernet1/29
>>>
>>>
>>> switch-core1# show routing hash x.x.x.x y.y.y.y
>>> Load-share parameters used for software forwarding:
>>> load-share mode: address source-destination port source-destination
>>> Hash for VRF "default"
>>> Hashing to path *y.y.y.y Eth1/29
>>> For route:
>>> y.y.y.0/24, ubest/mbest: 1/0
>>>     *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal
>>>
>>>
>>>
>>>
>>> From: NANOG <mailto:nanog-bounces+mhuff=ox.com at nanog.org> On Behalf Of
>>> Mike Hammett
>>> Sent: Monday, April 3, 2023 1:21 AM
>>> To: NANOG <mailto:nanog at nanog.org>
>>> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>>>
>>> We have a Nexus 3064 that is setup with partial BGP tables and is
>>> routing based on that.
>>>
>>>
>>> I've done a show ip bgp for an IP of interest and it has an expected
>>> next hop IP. I show ip arp on that next hop IP and it has the expected
>>> interface.
>>>
>>>
>>> However, sFlows show the packets leaving on a different interface, the
>>> one that would carry the default route for routes not otherwise known.
>>>
>>>
>>> If the next hop IP is expected and the ARP of that next hop IP is
>>> expected, why are packets leaving out an unexpected interface?
>>>
>>>
>>> -----
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>>
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20230404/5e080084/attachment.html>