<div dir="ltr">Not true, Intel Xeon E5 v3/4 have many more lanes than that. For example, this one has 40 <a href="https://ark.intel.com/content/www/us/en/ark/products/91754/intel-xeon-processor-e5-2680-v4-35m-cache-2-40-ghz.html">https://ark.intel.com/content/www/us/en/ark/products/91754/intel-xeon-processor-e5-2680-v4-35m-cache-2-40-ghz.html</a><div><br></div><div>Put two in the system and thats 80 lanes of PCIe Gen3.</div><div><br></div><div>Even newer procs in the bronze/silver/gold/platinum lineup have more lanes.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 24, 2020 at 8:57 AM Keith Medcalf <<a href="mailto:kmedcalf@dessus.com">kmedcalf@dessus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
And do not use an Intel CPU. <br>
<br>
Intel only has 4x PCIe lanes that are shared out into whatever configuration they claim to have and are totally unsuitable for use in a computer that actually has to be able to do high-speed I/O.<br>
<br>
-- <br>
Be decisive. Make a decision, right or wrong. The road of life is paved with flat squirrels who could not make a decision.<br>
<br>
>-----Original Message-----<br>
>From: NANOG <nanog-bounces+kmedcalf=<a href="mailto:dessus.com@nanog.org" target="_blank">dessus.com@nanog.org</a>> On Behalf Of<br>
>Eric Kuhnke<br>
>Sent: Saturday, 24 October, 2020 06:22<br>
>To: Jared Geiger <<a href="mailto:jared@compuwizz.net" target="_blank">jared@compuwizz.net</a>><br>
>Cc: NANOG <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>><br>
>Subject: Re: Linux router network cards<br>
><br>
>In addition to Jared's advice, I would recommend calculating PCI-Express<br>
>bandwidth bus points for whatever platform one is using.<br>
><br>
><br>
>For instance using the Intel X710-DA4, which could be capable in a<br>
>maximal scenario of 80Gbps of traffic, ensure it's in at least a PCI-E<br>
>3.0 x4 slot. And calculate the total number of PCI-E 3.0 x1 (or PCI-E 4.0<br>
>if a very new system) lanes that exist and are connected to the CPU. Big<br>
>difference between some options for Ryzen and Threadripper vs Intel CPUs,<br>
>towards the lower end of the cost range.<br>
><br>
><br>
>With recent Linux kernels if you have an Intel 510 or 710 series two or<br>
>four port card in a slot that can't support its full capability, you'll<br>
>get a warning in dmesg at boot time.<br>
><br>
><br>
><br>
>On Thu, Oct 22, 2020 at 9:30 PM Jared Geiger <<a href="mailto:jared@compuwizz.net" target="_blank">jared@compuwizz.net</a><br>
><mailto:<a href="mailto:jared@compuwizz.net" target="_blank">jared@compuwizz.net</a>> > wrote:<br>
><br>
><br>
> I use DANOS with Intel XL710 10G NICs in DPDK mode for linux based<br>
>routing.<br>
><br>
> If you're doing routing protocols, allocate 2 CPU cores to the<br>
>control plane and then a CPU core per 10G/1G interface for the dataplane,<br>
>plus an extra core for good measure. So for a 4 x 10G router taking in<br>
>full routes, 2 cores for control plane, 5 cores for the dataplane. Those<br>
>cores should be Intel Xeon E5-2600v3/4 or newer and faster the clocks,<br>
>the better.<br>
><br>
> Similar CPU core allocations if you choose TNSR.<br>
><br>
> On Thu, Oct 22, 2020 at 3:21 PM Jean St-Laurent via NANOG<br>
><<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a> <mailto:<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>> > wrote:<br>
><br>
><br>
> Chelsio cards are probably what you are looking for.<br>
><br>
> <a href="https://www.chelsio.com/terminator-6-asic/" rel="noreferrer" target="_blank">https://www.chelsio.com/terminator-6-asic/</a><br>
><br>
> It's closer to an asic than a traditional nic as the<br>
>router/firewall rules<br>
> are pushed directly into the hardware.<br>
><br>
> I don't know how good they are with linux and they seem to be<br>
>compatible.<br>
> <a href="https://www.chelsio.com/linux/" rel="noreferrer" target="_blank">https://www.chelsio.com/linux/</a><br>
><br>
> You will need to mess around a bit and fiddle here and there.<br>
>If you don't<br>
> mind using FreeBSD instead of linux, you could achieve a<br>
>smoother and more<br>
> integrated experience.<br>
><br>
> Jean<br>
><br>
> -----Original Message-----<br>
> From: NANOG <nanog-bounces+jean=<a href="mailto:ddostest.me@nanog.org" target="_blank">ddostest.me@nanog.org</a><br>
><mailto:<a href="mailto:ddostest.me@nanog.org" target="_blank">ddostest.me@nanog.org</a>> > On Behalf Of micah<br>
> anderson<br>
> Sent: Thursday, October 22, 2020 5:31 PM<br>
> To: Philip Loenneker <<a href="mailto:Philip.Loenneker@tasmanet.com.au" target="_blank">Philip.Loenneker@tasmanet.com.au</a><br>
><mailto:<a href="mailto:Philip.Loenneker@tasmanet.com.au" target="_blank">Philip.Loenneker@tasmanet.com.au</a>> >; NANOG<br>
> <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a> <mailto:<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>> ><br>
> Subject: RE: Linux router network cards<br>
><br>
><br>
> Thanks for the reply.<br>
><br>
> Philip Loenneker <<a href="mailto:Philip.Loenneker@tasmanet.com.au" target="_blank">Philip.Loenneker@tasmanet.com.au</a><br>
><mailto:<a href="mailto:Philip.Loenneker@tasmanet.com.au" target="_blank">Philip.Loenneker@tasmanet.com.au</a>> > writes:<br>
> > Take a look at the Mellanox ConnectX 5 series of cards. They<br>
>handle<br>
> > DPDK, PVRDMA (basically SR-IOV that allows live migration<br>
>between<br>
> > hosts), and can even process packets within the NIC for some<br>
><br>
> From what I can tell, SR-IOV/PVRDMA aren't really useful for<br>
>me in building<br>
> a router that wont be doing any virtualization.<br>
><br>
> If the card can do DPDK, can it do XDP?<br>
><br>
> > The slidedeck for the presentation is here:<br>
> > <a href="https://www.ausnog.net/sites/default/files/ausnog-" rel="noreferrer" target="_blank">https://www.ausnog.net/sites/default/files/ausnog-</a><br>
>2019/presentations/1<br>
> > .9_Rhod_Brown_AusNOG2019.pdf<br>
> ><br>
> > It's heavily targeting virtualised workloads but some of the<br>
>feature sets<br>
> apply to bare-metal uses too.<br>
><br>
> Yeah, this wont be a virtualized environment, just a router<br>
>passing packets,<br>
> dropping them, handling bgp and collecting flows.<br>
><br>
> --<br>
> micah<br>
><br>
><br>
<br>
<br>
<br>
<br>
</blockquote></div>