High throughput bgp links using gentoo + stipped kernel

Sun May 19 15:48:17 UTC 2013

> This is some fairly ancient hardware, so what you can get out if it will
> be limited. Though gige should not be impossible.
>

Agreed!!!

> The usual tricks are to make sure netfilter is not loaded, especially
> the conntrack/nat based parts as that will inspect every flow for state
> information. Either make sure those parts are compiled out or the
> modules/code never loads.
>
> If you have any iptables/netfilter rules, make sure they are 1)
> stateless 2) properly organized (cant just throw everything into FORWARD
> and expect it to be performant).
>

We do use a statefull iptables on our router, some forward rules...
This is known to be on of our issues, not sure if having a separate
iptables box would be the best and only solution for this?

> You could try setting IRQ affinity so both ports run on the same core,
> however I'm not sure if that will help much as its still the same cache
> and distance to memory. On modern NICS you can do tricks like tie rx of
> port 1 with tx of port 2. Probably not on that generation though.

Those figures include IRQ affinity tweaks at the kernel and APIC level.

>
> The 82571EB and 82573E is, while old, PCIe hardware, there should not be
> any PCI bottlenecks, even with you having to bounce off that stone age
> FSB that old CPU has. Not sure well that generation intel NIC silicon
> does linerate easily though.
>
> But really you should get some newerish hardware with on-cpu PCIe and
> memory controllers (and preferably QPI). That architectural jump really
> upped the networking throughput of commodity hardware, probably by
> orders of magnitude (people were doing 40Gbps routing using standard
> Linux 5 years ago).

Any ideas of the setup??? Maybe as far as naming some chipset, interface?
And xserver that is the best candidate. Will google.. :)

> Curious about vmstat output during saturation, and kernel version too.
> IPv4 routing changed significantly recently and IPv6 routing performance
> also improved somewhat.
>
>

Will get that output during peak on monday for you guys. Newest kernel
3.6 or 7...

Thank you so much for your insight,

Nick.