advice on BGP + CARP setup on FreeBSD

Tue Nov 5 22:50:24 UTC 2013

On 05/11/2013 19:31, Antoine Beaupré wrote:
> Our requirements:
> 
>  * free software, as much as possible
>  * inexpensive
>  * using existing operating system expertise (FreeBSD or Debian)

You need to make a decision on how to spend your money: on commodity router
hardware where you can easily get support if there's a problem, or on more
FOSS operating systems with a routing layer (e.g. openbgpd / bird / quagga
/ etc).

> As you can see, the uplinks are connected directly into a switch, in two
> separate VLANs. The reason for this is we want to be able to hotswap the
> routers in case of a hardware failure, but we have understood from
> Cogent's documentation that this is not a good practice because the
> links appears up even if the router goes down. What is your opinion on
> this?

Cogent is correct and their reasoning is correct.

> However, this seems to be a fairly exotic platform, most people running
> BGP with Cisco, Juniper or, in some cases Quagga or Bird for Linux
> machines. Are there recmomendations on using OpenBGP in production? Good
> / bad experiences? How many people are running Linux routers vs
> dedicated Cisco/Juniper/etc routers?

I run lots of different routing systems for a lot of different situations
(am currently using quagga, bird, openbgpd, cisco ios, cisco xr, junos and
brocade ironware for bgp stuff).  For small setups, it really doesn't make
a whole lot of difference so long as you run with a configuration which
supports both ibgp and an interior routing protocol like ospf or isis.
It's not going to make a whole lot of difference to you whether you use
quagga, openbgpd or bird because you're not going to stress the RIB engine
with only two providers.

Usually, it's better to run COTS routers (e.g. juniper / cisco / etc).  If
you don't want to do this, you will probably end up spending roughly the
same in terms of manpower, so don't be tempted to think that you're going
to save a whole lot with a free unix based system.

If you want a FOSS system and you have no preconceptions about routing, I'd
suggest using linux/freebsd + bird because bird is a truly wonderful RIB
engine.  If you are already familiar with cisco syntax, linux/freebsd +
quagga will do the job just fine.  If you have decided that you like
openbgpd and want all the features of openbgpd (including md5 passwords),
then you need openbsd + opengpd + openospfd, all of which I have found to
be frankly a pain to operate and maintain, although I think openbsd has
improved since the last time I used it in anger which was 3-4 years ago.

> Finally, we are likely to complete this setup with a CARP (the free
> equivalent of VRRP) on the inside of the network. FreeBSD can apparently
> group interfaces and communicate with OpenBGPd - did anyone deploy such
> a thing here? What are your experiences or advice?

linux carp is hopeless and I would strongly advise not to use linux if you
want to implement vrrp / carp.  Incidentally if anyone feels this is
unfair, they need to take a long hard look at the linux vmac implementation
and if they don't run screaming, I'll take my hat off.

The FreeBSD CARP implementation (which is borrow directly from openbsd)
usually works fine, but i've seen more than my fair share of kernel panics
on relatively recent freebsd relating to carp.

Srsly, get a cisco / juniper router.  Unless you're doing some incredibly
specialised large scale router implementation and you really know what
you're doing and why you're doing it, using a FOSS system will end up being
more expensive in terms of your time.

Nick