400G forwarding - how does it work?

Tue Jul 26 18:28:29 UTC 2022

On 25 July 2022 19:02:50 UTC, Saku Ytti <saku at ytti.fi> wrote:
>On Mon, 25 Jul 2022 at 21:51, James Bensley <jwbensley+nanog at gmail.com> wrote:
>
>> I have no frame of reference here, but in comparison to Gen 6 Trio of
>> NP5, that seems very high to me (to the point where I assume I am
>> wrong).
>
>No you are right, FP has much much more PPEs than Trio.

Can you give any examples?

>Why choose this NP design instead of Trio design, I don't know. I
>don't understand the upsides.

I think one use case is fixed latency. If you have minimal variation in your traffic you can provide a guaranteed upper bound on latency. This should be possible with the RTC model too of course, just harder because any variation in traffic at all, will result in a different run time duration, and I imagine it is easier to measure, find, and fix/tune chunks of code (running on separate cores, like in a pipeline) than in more code all running one core (like in RTC). So that's possibly a second benefit, maybe FP is easier to debug and measure changes?

>Downside is easy to understand, picture yourself as ucode developer,
>and you get task to 'add this magic feature in the ucode'.
>Implementing it in Trio seems trivial, add the code in ucode, rock on.
>On FP, you might have to go 'aww shit, I need to do this before PPE5
>but after PPE3 in the pipeline, but the instruction cost it adds isn't
>in the budget that I have in the PPE4, crap, now I need to shuffle
>around and figure out which PPE in line runs what function to keep the
>PPS we promise to customer.

That's why we have packet recirc <troll face>

>Let's look it from another vantage point, let's cook-up IPv6 header
>with crapton of EH, in Trio, PPE keeps churning it out, taking long
>time, but eventually it gets there or raises exception and gives up.
>Every other PPE in the box is fully available to perform work.
>Same thing in FP? You have HOLB, the PPEs in the line after thisPPE
>are not doing anything and can't do anything, until the PPE before in
>line is done.

This is exactly the benefit of FP vs NPI, less flexible, more throughput. NPU has served us (the industry) well at the edge, and FP is serving us well in the core.

>Today Cisco and Juniper do 'proper' CoPP, that is, they do ingressACL
>before and after lookup, before is normally needed for ingressACL but
>after lookup ingressACL is needed for CoPP (we only know after lookup
>if it is control-plane packet). Nokia doesn't do this at all, and I
>bet they can't do it, because if they'd add it in the core where it
>needs to be in line, total PPS would go down. as there is no budget
>for additional ACL. Instead all control-plane packets from ingressFP
>are sent to control plane FP, and inshallah we don't congest the
>connection there or it.

Interesting.

Cheers,
James.