<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.gmail-m4188415908271067832msoplaintext, li.gmail-m4188415908271067832msoplaintext, div.gmail-m4188415908271067832msoplaintext
{mso-style-name:gmail-m_4188415908271067832msoplaintext;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Nope, ASIC vendors are not ARM-based for PFE. Every “stage” is a very specialized ASIC with small programmability (not so small for P4 and some latest generation
ASICs).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ARM cores are for Network Processors (NP). ARM cores (with proper microcode) could emulate any “stage” of ASIC. It is the typical explanation for why NPs are
more flexible than ASIC.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Stages are connected to the common internal memory where enriched packet headers are stored. The pipeline is just the order of stages to process these internal
enriched headers.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The size of this internal header is the critical restriction of the ASIC, never disclosed or discussed (but people know it anyway for the most popular ASICs –
it is possible to google “key buffer”).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hint: the smallest one in the industry is 128bytes, the biggest 384bytes. It is not possible to process longer headers for one PFE pass.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Non-compressed SRv6 header could be 208bytes (+TCP/UDP +VLAN +L2 +ASIC_internal_staff). Hence, the need for compressed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It was a big marketing announcement from one famous ASIC vendor just a few years ago that some ASIC stages are capable of dynamically sharing common big external
memory (used for MAC/IP/Filters).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It may be internal memory too for small scalability, but typically it is external memory. This memory is always discussed in detail – it is needed for the operation
team.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It is only about headers. The packet itself (payload) is stored in the separate memory (buffer) that is not visible for pipeline stages.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">There were times when it was difficult to squeeze everything into one ASIC. Then one chip prepares an internal (enriched) header and may do some processing (some
simple stages), then send this header to the next chip for other “stages” (especially the complicated lookup with external memory connected). It is the artifact now.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Ed/<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Saku Ytti [mailto:saku@ytti.fi]
<br>
<b>Sent:</b> Tuesday, July 26, 2022 11:53 AM<br>
<b>To:</b> Vasilenko Eduard <vasilenko.eduard@huawei.com><br>
<b>Cc:</b> James Bensley <jwbensley+nanog@gmail.com>; NANOG <nanog@nanog.org><br>
<b>Subject:</b> Re: 400G forwarding - how does it work?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">On Tue, 26 Jul 2022 at 10:52, Vasilenko Eduard <<a href="mailto:vasilenko.eduard@huawei.com">vasilenko.eduard@huawei.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="gmail-m4188415908271067832msoplaintext">Juniper is pipeline-based too (like any ASIC). They just invented one special stage in 1996 for lookup (sequence search by nibble in the big external memory tree) – it was public information up to 2000year.
It is a different principle from TCAM search – performance is traded for flexibility/simplicity/cost.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">How do you define a pipeline? My understanding is that fabric and wan connections are in chip called MQ, 'head' of packet being some 320B or so (bit less on more modern Trio, didn't measure specifically)
is then sent to LU complex for lookup.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">LU then sprays packets to one of many PPE, but once packet hits PPE, it is processed until done, it doesn't jump to another PPE. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">Reordering will occur, which is later restored for flows, but outside flows reorder may remain.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">I don't know what the cores are, but I'm comfortable to bet money they are not ARM. I know Cisco used to ezchip in ASR9k but is now jumping to their own NPU called lightspeed, and lightspeed like
CRS-1 and ASR1k use tensilica cores, which are decidedly not ARM.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nokia, as mentioned, kind of has a pipeline, because a single packet hits every core in line, and each core does separate thing.<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Network Processors emulate stages on general-purpose ARM cores. It is a pipeline too (different cores for different functions, many cores for every function), just it is a virtual pipeline.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Ed/<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">-----Original Message-----<br>
From: NANOG [mailto:<a href="mailto:nanog-bounces%2Bvasilenko.eduard" target="_blank">nanog-bounces+vasilenko.eduard</a>=<a href="mailto:huawei.com@nanog.org" target="_blank">huawei.com@nanog.org</a>] On Behalf Of Saku Ytti<br>
Sent: Monday, July 25, 2022 10:03 PM<br>
To: James Bensley <<a href="mailto:jwbensley%2Bnanog@gmail.com" target="_blank">jwbensley+nanog@gmail.com</a>><br>
Cc: NANOG <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>><br>
Subject: Re: 400G forwarding - how does it work?<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">On Mon, 25 Jul 2022 at 21:51, James Bensley <<a href="mailto:jwbensley+nanog@gmail.com" target="_blank"><span style="color:windowtext;text-decoration:none">jwbensley+nanog@gmail.com</span></a>> wrote:<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">> I have no frame of reference here, but in comparison to Gen 6 Trio of
<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">> NP5, that seems very high to me (to the point where I assume I am
<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">> wrong).<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">No you are right, FP has much much more PPEs than Trio.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">For fair calculation, you compare how many lines FP has to PPEs in Trio. Because in Trio single PPE handles entire packet, and all PPEs run identical ucode, performing same work.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">In FP each PPE in line has its own function, like first PPE in line could be parsing the packet and extracting keys from it, second could be doing ingressACL, 3rd ingressQoS, 4th ingress lookup and so forth.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Why choose this NP design instead of Trio design, I don't know. I don't understand the upsides.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Downside is easy to understand, picture yourself as ucode developer, and you get task to 'add this magic feature in the ucode'.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Implementing it in Trio seems trivial, add the code in ucode, rock on.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">On FP, you might have to go 'aww shit, I need to do this before PPE5 but after PPE3 in the pipeline, but the instruction cost it adds isn't in the budget that I have in the PPE4, crap, now I need to shuffle
around and figure out which PPE in line runs what function to keep the PPS we promise to customer.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Let's look it from another vantage point, let's cook-up IPv6 header with crapton of EH, in Trio, PPE keeps churning it out, taking long time, but eventually it gets there or raises exception and gives up.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Every other PPE in the box is fully available to perform work.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Same thing in FP? You have HOLB, the PPEs in the line after thisPPE are not doing anything and can't do anything, until the PPE before in line is done.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">Today Cisco and Juniper do 'proper' CoPP, that is, they do ingressACL before and after lookup, before is normally needed for ingressACL but after lookup ingressACL is needed for CoPP (we only know after lookup
if it is control-plane packet). Nokia doesn't do this at all, and I bet they can't do it, because if they'd add it in the core where it needs to be in line, total PPS would go down. as there is no budget for additional ACL. Instead all control-plane packets
from ingressFP are sent to control plane FP, and inshallah we don't congest the connection there or it.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">> Cheers,<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">> James.<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> <o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext">--<o:p></o:p></p>
<p class="gmail-m4188415908271067832msoplaintext"> ++ytti<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal"> ++ytti<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>