2006.06.06 NANOG-NOTES DNS reflector attacks

Wed Jun 7 10:30:25 UTC 2006

(I was going to try to get all the notes from today's panels out
before going to bed, but I fell asleep on my keyboard finishing
up these notes, so I think I'm going to wait and send the batch
of Tuesday and Wednesday notes out after things wrap up on
Wednesday.  Sorry about the delay, but I need a bit more sleep
I think.  ^_^;;  --Matt)

2006.06.06 Morning welcome, and introduction
of Chris Morrow, panelist

Please fill out survey today if you're going
to be leaving!

Frank Scalzo, Verisign
Recent DNS reflector attacks.

Attacker breaks into innocent authoritative
DNS server, publishes large text record;
then does queries from zombie army
against that record, with sources spoofed
with victim IP.

5 gig attack, 2.2G made it, 3gig didn't.

E.TN.CO.ZA DNS attack, 64 byte query,
63:1 amplification, 4028 byte answer
34,668 reflectors.

Victim sees 5G of traffic, 144,142bps
per reflector, 13.5packets per second
4.5DNS answers per second.

reflectors won't see this as anomolous for
the most part; top talker only sent 8.5
answers per second.

No visibility into the attacker at all,
but best guess was 79Mb of source generated
5GB of responses.
Record was maliciously installed;
2 auth servers, 1 compromised; 65% response,
35% name error.

Answer comes in 3 fragments, larger than
normal MTU.
Attack came in 3 phases.
first port 666, then port 53 and 666, then all 53.
Port shifts are nearly instant, so fast command
and control system in place for it.

Filter out open recursive DNS servers;
you can't put ACL in for 500,000 DNS
servers.
What about limiting DNS packets to 512 bytes?
will break things.
What about blocking 53 outside of your network
hierarchy, force people to use your resolvers?

What about discarding fragments?

Challenge is getting your upstream to implement
it, unless you have hardware and pipes to handle
the flood coming at you to start with.
Some ISPs won't do it unless they see live attack
traffic, and a 24 minute attack is too short lived
for ISPs to see and react to.

data from Jan 11 - Feb 27 this year.
Attack queries/second consistent with avg reflector qps.

one reflector sent 1.9M DNS answers to 1593 victims,
605 different queries to generate answers.
180TB of attack traffic on Feb 1st.
after feb 15th, ramped down.

Assume 4KB response packet,
see attacks between 3G and 7G, the scary part is
that it only took 130Mb to generate the 7G attack,
and the 3 gig attacks are all from less than a
fastE connected compromised web server.

500,000 reflectors with 2G source could generate
a 120GB DoS attack.

Top victim got over 130Tb of attack traffic, top bunch
are all over 100Tb

65,461 ports used, Top port is less than 10% of
traffic though

top 20 domains used, mostly innocent bystanders.

Internet root . was second highest domain used;
certainly can't filter *that* out.

Fundamental challenge;
UDP lacks 3 way handshake, easy to spoof
DNS is easy target, so many unsecured DNS servers
Other UDP servers need to be evaluated as well

DNS
closing 500,000 open recursive DNS servers will
be very, very painful.
poor separation between authoritative and recursive
DNS servers.
BIND allow-query ACLs, recursive DNS servers should
not accept queries from outside.

What if it's an embedded system like a wireless
gateway?

We depend on large records for DNSSec, etc.

Beyond open recursive DNS servers
root domain "." was used
most authoritative name servers will answer with an
upward referral
 doesn't include actual IPs, but it's still 438 bytes,
 and pretty much every DNS server responds to it.

Source validation
 IETF BCP 38
 How do you manage 70,000 ACLs on 500 routers?
 what about people who are multi-homed with static routes?
 what about legacy stuff that works but shouldn't?
 strict RPF breaks with traffic asymmetry; loose RPF
  doesn't help with this.
 ISPs see the problem as long, hard, expensive to
  overcome, and they're right.
 If we never start trying, we'll never fix it!

Close open recursive DNS servers
DNS servers should include filtering
SOHO router vendors should fix their DNS proxy code,
 don't listen on outside interface
BCP 38
 otherwise we'll be jumping from protocol to protocol.

Questions?
Q: What does verisign do to protect their DNS servers?
A: Anycast, massive peering and transit capacity

Q: Jared Mauch, NTT/America; he turned on unicast rpf
on the NANOG upstream link.  372,000 packets that
people here have sent failed the RPF check.
BCP 38 is hard
Paul Quinn asked what percentage of the traffic that
is.
Bora Akyul, Broadcomm--any data on source ranges
on the packets being seen?
He could look at the 1 in 10,000 netflow sampling
to see, but the individual link is a /30, looks
like a normal customer link.
The Merit router isn't RPF'ing either.

Q: Ren Provo asks when they will peer;
A: not yet, next few months,
Miami Terremark, and other sites domestically
and internationally in next year and a half.