IPv4 ANYCAST setup

Mark Smith nanog at 85d5b20a518b8f6864949bd940457dc124746ddc.nosense.org
Fri Mar 26 23:37:49 UTC 2010


On Fri, 26 Mar 2010 14:24:21 +0100
Jeroen Massar <jeroen at unfix.org> wrote:

> InterNetX - Lutz Muehlig wrote:
> > Hello,
> > 
> > has someone experience in anycast ipv4 networks (to support DNS)?
> 
> "Never been done" "Dangerous" "TCP does not work" etc etc etc.
> 
> I assume quite a number of people know how to do it, especially as
> several root DNS servers abuse it.
> 
> Simple recipe:
>  - Box with:
>    - Your favourite OS
>    - Quagga or OpenBGPd
>    - Your favourite DNS server
>  - Announce the IP of the anycast node in BGP
>  - Monitor the DNS server, when it does not work kill your local BGPd
>    and notify the admins that it broke
> 
> That is it. Probably with the above couple of things, google a bit and
> find the rest.
> 

I was involved in building an anycast setup where we had two anycast
DNS /32 addresses. Each of them was the backup for the other i.e. each
DNS server was announcing both /32s via BGP, with opposite weights. If
one failed, the other DNS server then took over the failed DNS
cache's queries, and as it was also already an operational DNS
server for one of the anycast addresses, it's DNS cache was already hot.

For load balancing, we alternated the order of announcing the DNS
server addresses in e.g. PPP IPCP, DHCP. That worked somewhat
surprisingly well - the peak query per second values on each of them
were no more than about 10% different.

One trap we got caught by was stateful firewalling on the host. We knew
to up the number of stateful connections, however on that particular
Linux distro there were two places it was set - /etc/sysctl.conf and in
the iptables configuration. We only knew about the first, so when the
firewall rules were updated the number of supported stateful
connections was dropped down to too low a level. It wasn't funny to
have one DNS server stop answering queries, and have it's own
monitoring script fail itself, switch all the traffic to the other one
and then have that die too for the same reason. Lots of gnashing of
teeth until we worked out .

The final and better solution was to stop doing stateful firewalling on
DNS queries, using the iptables 'raw' table.

Regards,
Mark.




More information about the NANOG mailing list