DNS problems to RoadRunner - tcp vs udp

Sun Jun 15 13:02:49 UTC 2008

> There is no call for insults on this list - Rather thought this list was 
> about techincal discussions affecting all of us and keeping DNS alive 
> for the majority of our customers certainly qualifies.
> 
> We/I am more than aware of the DNS mechanisms and WHY there are there 
> trouble is NO DNS server can handle directed TCP attacks even the root 
> servers crumbled under directed botnet activity and we have taken the 
> decision to accept some collateral damage in order to keep services 
> available.     We are a well connected university network with 
> multi-gigabit ingress and egress with 10G on Abilene  so we try to 
> protect the internet from attacks originating within our borders AND we 
> really feel the full wrath of botnets as we do not have a relatively 
> slow WAN link to buffer the effects.
> 
> Yes - we are blocking TCP too many problems with drone armies and we 
> started about a year ago when our DNS servers became unresponsive for no 
> apparent reason.   Investigation showed TCP flows of hundreds of 
> megabits/sec and connection table overflows from tens of thousands of 
> bots all trying to simultaneously do zone transfers and failing tried 
> active denial systems and shunning with limited effectiveness.
> 
> We are well aware of the host based mechanisms to control zone 
> information,  Trouble is with TCP if you can open the connection you can 
> DoS so we don't allow the connection to be opened and this is enforced 
> at the network level where we can drop at wire speed.     Open to better 
> ideas but if you look at the domain in my email address you will see we 
> are a target for hostile activity just so someone can 'make their bones'.
> 
> Also recall we have a comittment to openess so we would like to make TCP 
> services available but until we have effective DNS DoS mitigation which 
> can work with 10Gb links It's not going to happen. 

This could be a real problem.  Of course, one solution is to take a
shotgun, point it at your foot, and pull the trigger.  As you noted, a
long history of operational experience suggests that most resolver
traffic is UDP, which means that you won't have that many problems from
doing this.  However, you can kiss that "5 nines of availability" you
claim to be interested in goodbye.

Nathan Ward and Sean Donelan have covered most of the points I would have
made had I written this message earlier.  I will also point out, however,
that you don't even necessarily need a load balancer to do the tcp/53
screening, but there are a lot of neat and clever things that you could
do (some depending on with or without).

Given the relatively low level of usage for TCP DNS lookups, plus the fact
that it ought to be trivial to automatically identify hosts that are doing
TCP things that they shouldn't be doing (one zone transfer request might
be a reasonable thing, as many old-timers might do that sort of thing, but
won't try repeatedly if the server refuses), it kind of mystifies me as to
why you seem to imply that this is so difficult.

For example, it only took a few minutes to come up with this, for FreeBSD,
which takes advantage of the radix table support in ipfw.

#! /bin/sh -

grep -i axfr /var/log/messages | grep "zone transfer .* denied" | (
        while read a b c d e f g h i j k l m n; do
                g=`echo "${g}" | sed 's:#.*::'`
                # Maybe do some other processing ... a bad example: log it
                # echo noticed "${g}" "${j}" | logger -p info -t protector
                # Put out the IP's we want to consider filtering
                echo "${g}"
        done
) | sort | uniq -c | (
        while read a b; do
                if [ "${a}" -gt 3 ]; then
                        2> /dev/null ipfw table 1 add "${b}"/32 && (echo banned "${b}" "${a}" | logger -p warning -t protector)
                fi
        done
)

Needs to have a rule like this installed in the system firewall table:

% ipfw add [nnn] deny tcp from table\(1\) to me 53

I think a real solution would be more sophisticated than this, but it's a
starting point.  This sort of thing limits "collateral damage" to hosts
that have attempted an unreasonable number of zone transfers, which is
still unpleasant, but is less likely to break legitimate uses.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.