Advice requested for OpenBSD vs. Linux/OpenBGP vs. Quagga router deployment.

Sat Dec 20 10:49:26 UTC 2008

Hi Marc,

> We are a software development firm that currently delivers our install ISOs via Sourceforge.  We need to start serving them ourselves for marketing reasons and are therefore increasing our bandwidth and getting a 2nd ISP in our datacenter.  Both ISPs will be delivering 100mbit/sec links.  We don't expect to increase that for the next year or so and expect average traffic to be about 40-60mbit/sec.
> 
> We are planning to run two OpenBSD based firewalls (with CARP and pf) running OpenBGP in order to connect to the two ISPs.
> 
> I saw from previous email that Quagga was recommended as opposed to OpenBGP.  Any further comments on that?  Also,  any comments on the choice of OpenBSD vs. Linux?

I would suggset checking out Vyatta Linux as a possible Linux solution.
It's designed to be configured as a routing/firewall platform. One caveat,
I have never used it but it seems to be mentioned in this list from time
to time.

Now for my rant.

I attempted a setup as you describe using two servers using pf, carp,
and openbgp. I also had VLANs configured (each VLAN interface had it's
own CARP interface). I tried both load-balanced and failover mode but
the results weren't desirable.

The routers were connected to a switch which connects the servers and
the ISP connection. There was only one drop from the ISP but each router
had it's own /30 and BGP session on it's own VLAN. The remaining servers
were also VLANned appropriately. Each VLAN interface on the router that
connects to the servers would also have an accompanying CARP interface.

There were a myriad of problems when attempting my setup. These are some
that I distinctly recall.

* In load-balancing mode I would unplug a router. The other router would
register as a CARP master but didn't forward the remaining traffic.

* In failover mode when unplugging a router the other router would forward
traffic for certain VLANs and wouldn't register as master for the others.

In hindsight I should've reached out to the openbsd community for
assistance. It's possible I was running into bugs in the CARP code or
I was simply doing it all wrong.  However I was under a time crunch and
this was merely a favour for a friend in need. I didn't want to further
disrupt the network by testing so I ended up going with a single router
setup (still openbsd though). I haven't revisited the daul router setup
since everything has been working fine and dandy with one router.

Regardless of what OS choice you make be sure to thoroughly test your
network setup and make sure it works as planned. Lastly don't hesitate
to ask the appropriate people for help. You may have discovered oddities
that noone else has.

Good luck,
Naveen