High throughput bgp links using gentoo + stipped kernel

Nick Khamis symack at gmail.com
Sun May 19 15:48:17 UTC 2013

> This is some fairly ancient hardware, so what you can get out if it will
> be limited. Though gige should not be impossible.


> The usual tricks are to make sure netfilter is not loaded, especially
> the conntrack/nat based parts as that will inspect every flow for state
> information. Either make sure those parts are compiled out or the
> modules/code never loads.
> If you have any iptables/netfilter rules, make sure they are 1)
> stateless 2) properly organized (cant just throw everything into FORWARD
> and expect it to be performant).

We do use a statefull iptables on our router, some forward rules...
This is known to be on of our issues, not sure if having a separate
iptables box would be the best and only solution for this?

> You could try setting IRQ affinity so both ports run on the same core,
> however I'm not sure if that will help much as its still the same cache
> and distance to memory. On modern NICS you can do tricks like tie rx of
> port 1 with tx of port 2. Probably not on that generation though.

Those figures include IRQ affinity tweaks at the kernel and APIC level.

> The 82571EB and 82573E is, while old, PCIe hardware, there should not be
> any PCI bottlenecks, even with you having to bounce off that stone age
> FSB that old CPU has. Not sure well that generation intel NIC silicon
> does linerate easily though.
> But really you should get some newerish hardware with on-cpu PCIe and
> memory controllers (and preferably QPI). That architectural jump really
> upped the networking throughput of commodity hardware, probably by
> orders of magnitude (people were doing 40Gbps routing using standard
> Linux 5 years ago).

Any ideas of the setup??? Maybe as far as naming some chipset, interface?
And xserver that is the best candidate. Will google.. :)

> Curious about vmstat output during saturation, and kernel version too.
> IPv4 routing changed significantly recently and IPv6 routing performance
> also improved somewhat.

Will get that output during peak on monday for you guys. Newest kernel
3.6 or 7...

Thank you so much for your insight,


More information about the NANOG mailing list