DNS problems to RoadRunner - tcp vs udp

Jeroen Massar jeroen at unfix.org
Sat Jun 14 16:54:49 CDT 2008


Scott McGrath wrote:
> 
> There is no call for insults on this list

Insults? Where? If you feel insulted by any of the comments made on this 
list by people, then you probably are indeed on the wrong list. But that 
is just me.

> - Rather thought this list was 
> about techincal discussions affecting all of us and keeping DNS alive 
> for the majority of our customers certainly qualifies.

[..blabber about DNS attacks over TCP..]

If I where a botnet herder and I had to take out your site and I was 
going to pick TCP for some magical reason then I would not care about 
your DNS servers, I would just hit your webservers, hard. I mean just 
the 'index.html' (http://www.harvard.edu/) is 24Kb, that is excluding 
pictures and there is bound to be larger data there which you are going 
to send and the bots only have to say "ACK" to once in a while.

Multiply that by say a small botnet of 1M hosts, each just requests that 
24Kb file. You will have a million flows and won't have any way to rate 
limit that or control it. Your link was already full trying to send it 
back to the clients and next to that your server was probably not able 
to process it in the first place. Simple, effective, nothing you can do 
about it, except get way and way more hardware.

If somebody wants to take you out, they will take you out. Just get one 
other box with 10GE (not too hard to do) or just get a million of them 
with a little bit of connectivity (which is quite easy apparently)...

> We/I am more than aware of the DNS mechanisms and WHY there are there 
> trouble is NO DNS server can handle directed TCP attacks even the root 
> servers crumbled under directed botnet activity and we have taken the 
> decision to accept some collateral damage in order to keep services 
> available.

"The root servers crumbled" wow, I must have missed somebody taking out 
all the 13 separate and then individually anycasted root servers. Which 
btw only do UDP as currently '.' is still small enough.

$ dig @a.root-servers.net. . NS +tcp
[..]
;; Query time: 95 msec
;; SERVER: 2001:503:ba3e::2:30#53(2001:503:ba3e::2:30)
;; WHEN: Sat Jun 14 23:45:52 2008
;; MSG SIZE  rcvd: 604

That is only 1 packet to 1 packet, still only 500 bytes. While your 
little webserver would generate 24kb for that same sequence.

>    We are a well connected university network with 
> multi-gigabit ingress and egress with 10G on Abilene  so we try to
> protect the internet from attacks originating within our borders AND we 
> really feel the full wrath of botnets as we do not have a relatively 
> slow WAN link to buffer the effects.

The whole point generally of botnets is just the Denial of Service 
(DoS), if that is because your link is full or the upstreams link is 
full or because the service can't service clients anymore.

But clearly, as you are blocking TCP-DNS you are DoSing yourself 
already, so the botherders win.

Also note that Abilene internally might be 10G and in quite some places 
even 40G, but you still have to hand it off to the rest of the world and 
those will count as those 'slow WAN' links that you think everybody else 
on this planet is behind. (Hint: 10GE is kinda the minimum for most 
reasonably sized ISP's)

> Yes - we are blocking TCP too many problems with drone armies and we 
> started about a year ago when our DNS servers became unresponsive for no 
> apparent reason.   Investigation showed TCP flows of hundreds of 
> megabits/sec and connection table overflows from tens of thousands of 
> bots all trying to simultaneously do zone transfers and failing tried 
> active denial systems and shunning with limited effectiveness.

How is a failed AXFR going to generate a lot of traffic, unless they are 
repeating themselves over and over and over again? Thus effectively just 
packeting you?

Also, are you talking about Recursive or Authoritive DNS servers here?
Where those bots on your network, or where they remote?

> We are well aware of the host based mechanisms to control zone 
> information,  Trouble is with TCP if you can open the connection you can 
> DoS so we don't allow the connection to be opened and this is enforced 
> at the network level where we can drop at wire speed.

Do you mean that the hosts which do TCP are allowed to do transfers or 
not? As in the latter case they can't generate big answers, they just 
get 1 packet back and then end then FIN.

Note also, that if they are simply trying to overload your hosts, UDP is 
much more effective in doing that already and you have that hole wide 
open apparently otherwise you wouldn't have DNS.

> Open to better 
> ideas but if you look at the domain in my email address you will see we 
> are a target for hostile activity just so someone can 'make their bones'.

It probably has nothing to do with the domain name, it more likely has 
something to do with certain services that are available or provided on 
your network.

> Also recall we have a comittment to openess so we would like to make TCP 
> services available but until we have effective DNS DoS mitigation which 
> can work with 10Gb links It's not going to happen.

You think that 10Gb is a 'fat link', amusing ;)

There are various vendors, most likely also reading on this list, who 
can be more than helpful in providing you with all kinds of bad, but 
also a couple of good solutions to most networking issues that you are 
apparently having.

But the biggest issue you seem to have is not knowing what the DoS 
kiddies want to take out and why they want to take it out.

Greets,
  Jeroen

PS: You do know that an "NS" record is not allowed to point to a CNAME I 
hope? (NS3.harvard.edu CNAME ns3.br.harvard.edu. RFC1912 2.4 ;)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 187 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20080614/29473ad3/attachment.bin>


More information about the NANOG mailing list