High throughput bgp links using gentoo + stipped kernel

Sun May 19 23:23:14 UTC 2013

On Sat, May 18, 2013 at 11:39:55AM -0400, Nick Khamis wrote:
> Hello Everyone,
> 
> We are running:
> 
> Gentoo Server on Dual Core Intel Xeon 3060, 2 Gb Ram
> Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
> Controller (rev 06)
> Ethernet controller: Intel Corporation 82573E Gigabit Ethernet
> Controller (rev 03)
> 
> 2 bgp links from different providers using quagga, iptables etc....
> 
> We are transmitting an average of 700Mbps with packet sizes upwards of
> 900-1000 bytes when the traffic graph begins to flatten. We also start
> experiencing some crashes at that point, and not have been able to
> pinpoint that either.
> 
> I was hoping to get some feedback on what else we can strip from the
> kernel. If you have a similar setup for a stable platform the .config
> would be great!
> 
> Also, what are your thoughts on migrating to OpenBSD and bgpd, not
> sure if there would be a performance increase, but the security would
> be even more stronger?

That hardware should be fine to do two gig ports upstream, with another
two to go to your network?

I'd check with "vmstat 1" to see what your interrupt rate is like, if it's
above 40k/sec I'd check coalescing settings.

I also prefer OpenBSD/OpenBGP myself.  It's a simpler configuration, with less
things to "fix".

With Linux you have to disable reverse path filtering, screw around with iptables
to do bypass on stateful filtering.  Then Quagga itself can be buggy. (my original
reason for shifting away from Linux was that Quagga didn't fix enough of Zebra's
bugs.. although that was many years ago, things may have improved a little by then,
but ime significantly buggy software tends to stay buggy even with fixing)

With regards to security of OpenBSD versus Linux, you shouldn't be exposing any
services to the world with either.  And it's more stability/configuration that would
push me to OpenBSD rather than performance.

And with regards to crashing I'd try and figure out what was happening there quickly
before making radical changes.  Is it running out of memory, is Quagga dying?  Is
there a default route that works when Quagga crashes?  One issue I had was I found
Quagga crashing leaving a whole lot of routes lingering in the table, and I had a
script that'd go through and purge them.

I'm also a bit confused about your dual upstreams with two ethernet interfaces total,
are they both sharing one pipe, or are there some Broadcom or such ethernet interfaces
too.  I've found Broadcom chipsets can be a bit problematic, and the only stability
issue I've ever had with OpenBSD is a Broadcom interface wedging for minutes under DDOS
attack, which was gigabit'ish speed DDOS with older hardware than you.

oh, to check coalescing settings under linux use: "ethtool -c eth0; ethtool -c eth1"

Ben.