NANOG Digest, Vol 5, Issue 33

Paul Dowles pdowles at cogentco.com
Sun Jun 15 08:12:32 UTC 2008


 
-----Original Message-----
From: nanog-request at nanog.org

Date: Sat, 14 Jun 2008 21:54:35 
To: <nanog at nanog.org>
Subject: NANOG Digest, Vol 5, Issue 33


Send NANOG mailing list submissions to
	nanog at nanog.org

To subscribe or unsubscribe via the World Wide Web, visit
	http://mailman.nanog.org/mailman/listinfo/nanog
or, via email, send a message with subject or body 'help' to
	nanog-request at nanog.org

You can reach the person managing the list at
	nanog-owner at nanog.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of NANOG digest..."


Today's Topics:

   1. RE: [NANOG] Introducing latency for testing? (Frank Bulk - iNAME)
   2. Re: .255 addresses still not usable after all these years?
      (Greg VILLAIN)
   3. Re: DNS problems to RoadRunner - tcp vs udp (Scott McGrath)
   4. Re: DNS problems to RoadRunner - tcp vs udp (Jeroen Massar)
   5. Re: [NANOG] Introducing latency for testing? (Chris Marlatt)
   6. Re: [NANOG] Introducing latency for testing? (Joel Jaeggli)
   7. Re: DNS problems to RoadRunner - tcp vs udp (Scott McGrath)
   8. Re: DNS problems to RoadRunner - tcp vs udp (Simon Leinen)
   9. Re: DNS problems to RoadRunner - tcp vs udp (Jeroen Massar)


----------------------------------------------------------------------

Message: 1
Date: Sat, 14 Jun 2008 13:34:53 -0500
From: "Frank Bulk - iNAME" <frnkblk at iname.com>
Subject: RE: [NANOG] Introducing latency for testing?
To: "'Mike Lyon'" <mike.lyon at gmail.com>, "NANOG" <nanog at merit.edu>
Message-ID:
	<!&!AAAAAAAAAAAuAAAAAAAAAKTyXRN5/+lGvU59a+P7CFMBAN6gY+ZG84BMpVQcAbDh1IQAAAATbSgAABAAAACMh0iCc5e/[email protected]>
	
Content-Type: text/plain;	charset="us-ascii"

It's not free, but at a recent trade show I did see what appeared to be an
affordable unit from Apposite Technologies (apposite-tech.com).  And there's
always PacketStorm.

Frank

-----Original Message-----
From: Mike Lyon [mailto:mike.lyon at gmail.com] 
Sent: Friday, May 02, 2008 3:13 PM
To: NANOG
Subject: [NANOG] Introducing latency for testing?

So I want to mimic some latency in a test network for DB replication.
I am wondering what other's have used for this? Obviously, the best
way to would be to actually have one box across the US or across the
globe to actually test against but what if you don't have that? Are
there any GPL software router solutions that would allow you to tweak
the latency in between the two test boxes?

Thanks in advance.

-Mike

_______________________________________________
NANOG mailing list
NANOG at nanog.org
http://mailman.nanog.org/mailman/listinfo/nanog




------------------------------

Message: 2
Date: Sat, 14 Jun 2008 20:33:21 +0200
From: Greg VILLAIN <nanog at grrrrreg.net>
Subject: Re: .255 addresses still not usable after all these years?
To: nanog at nanog.org
Message-ID: <A2776BAD-491D-4F3D-9645-02EC7492E1FB at grrrrreg.net>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

On Jun 14, 2008, at 12:26 AM, Mike Lewinski wrote:

> David Hubbard wrote:
>> I remember back in the day of old hardware and operating
>> systems we'd intentionally avoid using .255 IP addresses
>> for anything even when the netmask on our side would have
>> made it fine, so I just thought I'd try it out for kicks
>> today.  From two of four ISP's it worked fine, from Verizon
>> FIOS and Road Runner commercial, it didn't.  So I guess
>> that old problem still lingers?
>
> The TCP/IP stack in Windows XP is broken in this regard, possibly in  
> Vista as well, though I've yet to have the displeasure of finding  
> out. I have a router with a .255 loopback IP on it. My Windows XP  
> hosts cannot SSH to it. The specific error that Putty throws is  
> "Network error: Cannot assign requested address".
>
> At least if I ever need to completely protect a device from access  
> by Windows users, I have a good option :)
>
> Mike

 From what I recall, Microsoft's stack was based on the only free one  
they could afford back in the Trumpet/Winsock days, namely BSD's.
It is either dependent on how the stack is integrated, or it simply  
implies that BSD's stack is(was) also broken (I'd tend to doubt that).
Also, Vista's stack was supposed to have been re-developed from  
scratch, never checked it.

Greg VILLAIN






------------------------------

Message: 3
Date: Sat, 14 Jun 2008 16:40:10 -0400
From: Scott McGrath <mcgrath at fas.harvard.edu>
Subject: Re: DNS problems to RoadRunner - tcp vs udp
To: Randy Bush <randy at psg.com>
Cc: nanog at merit.edu
Message-ID: <48542CAA.5010503 at fas.harvard.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Not to toss flammables onto the pyre. 

BUT there is a large difference from what the RFC's allow and common 
practice.   In our shop TCP is blocked to all but authoratative 
secondaries as TCP is sinply too easy to DoS a DNS server with.   We 
simply don't need a few thousand drones clogging the TCP connection 
table all trying to do zone transfers ( yes it happened and logs show 
drones are still trying )

For a long time there has been a effective practice of

UDP == resolution requests
TCP == zone transfers

It would have been better if a separate port had been defined for zone 
transfers as that would obviate the need for a application layer gateway 
to allow TCP transfers so that zone transfers can be blocked and 
resolution requests allowed for now all TCP is blocked.

Now just because someone has a bright idea they drag out a 20 y/o RFC 
and say SEE, SEE you must allow this because the RFC says so all the 
while ignoring the 20 years of operational discipline
that RFC was written when the internet was like the quad at college 
everyone knew one and other and we were all working towards a common 
goal of interoperability and open systems ,   These days the net is more 
like a seedy waterfront after midnight where criminal gangs are waiting 
to ambush the unwary and consequently networks need to be operated from 
that standpoint.

At the University networking level it is extremely difficult as we need 
to maintain a open network as much as possible but protect our 
infrastructure services so that they have 5 nines of availability
back in the day a few small hosts would serve DNS nicely and we did  not 
have people trying to take them down and/or infecting local hosts and 
attempting DHCP starvation attacks.   And no we are not at the 5 nines 
level but we are working on it.

 
- Scott


Randy Bush wrote:
>> If my server responded to TCP queries from anyone other than a secondary
>> server, I would be VERY concerned.
>>     
>
> you may want to read the specs
>
> randy
>   




------------------------------

Message: 4
Date: Sat, 14 Jun 2008 22:47:47 +0200
From: Jeroen Massar <jeroen at unfix.org>
Subject: Re: DNS problems to RoadRunner - tcp vs udp
To: Scott McGrath <mcgrath at fas.harvard.edu>
Cc: nanog at merit.edu
Message-ID: <48542E73.9070109 at spaghetti.zurich.ibm.com>
Content-Type: text/plain; charset="iso-8859-1"

Scott McGrath wrote:
[..]
> For a long time there has been a effective practice of
> 
> UDP == resolution requests
> TCP == zone transfers

WRONG. TCP is there as a fallback when the answer of the question is too 
large. Zone transfer you can limit in your software. If you can't 
configure your dns servers properly then don't run DNS.
Also note that botnets have much more effective ways of taking you out.

And sometimes domains actually require TCP because there are too many 
records for a label eg http://stupid.domain.name/node/651
If you are thus blocking TCP for DNS resolution you suddenly where 
blocking google and thus for some people "The Internet".

Also see:
http://homepages.tesco.net/J.deBoynePollard/FGA/dns-edns0-and-firewalls.html

(Which was the second hit for google(EDNS0) after a link to RFC2671)

Greets,
  Jeroen

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 187 bytes
Desc: OpenPGP digital signature
Url : http://mailman.nanog.org/pipermail/nanog/attachments/20080614/8dc85bfe/attachment-0001.pgp 

------------------------------

Message: 5
Date: Sat, 14 Jun 2008 16:47:32 -0400
From: Chris Marlatt <cmarlatt at rxsec.com>
Subject: Re: [NANOG] Introducing latency for testing?
To: frnkblk at iname.com
Cc: NANOG <nanog at merit.edu>
Message-ID: <48542E64.3000704 at rxsec.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Frank Bulk - iNAME wrote:
> It's not free, but at a recent trade show I did see what appeared to be an
> affordable unit from Apposite Technologies (apposite-tech.com).  And there's
> always PacketStorm.
> 
> Frank
> 
> -----Original Message-----
> From: Mike Lyon [mailto:mike.lyon at gmail.com] 
> Sent: Friday, May 02, 2008 3:13 PM
> To: NANOG
> Subject: [NANOG] Introducing latency for testing?
> 
> So I want to mimic some latency in a test network for DB replication.
> I am wondering what other's have used for this? Obviously, the best
> way to would be to actually have one box across the US or across the
> globe to actually test against but what if you don't have that? Are
> there any GPL software router solutions that would allow you to tweak
> the latency in between the two test boxes?
> 
> Thanks in advance.
> 
> -Mike
> 
> _______________________________________________
> NANOG mailing list
> NANOG at nanog.org
> http://mailman.nanog.org/mailman/listinfo/nanog
> 
> 

IIRC ipfw can do this using dummynet and the delay directive.

Regards,

	Chris



------------------------------

Message: 6
Date: Sat, 14 Jun 2008 14:08:25 -0700
From: Joel Jaeggli <joelja at bogus.com>
Subject: Re: [NANOG] Introducing latency for testing?
To: Chris Marlatt <cmarlatt at rxsec.com>
Cc: NANOG <nanog at merit.edu>
Message-ID: <48543349.7060400 at bogus.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Chris Marlatt wrote:
> Frank Bulk - iNAME wrote:
>> It's not free, but at a recent trade show I did see what appeared to 
>> be an
>> affordable unit from Apposite Technologies (apposite-tech.com).  And 
>> there's
>> always PacketStorm.
>>
>> Frank
>>
>> -----Original Message-----
>> From: Mike Lyon [mailto:mike.lyon at gmail.com] Sent: Friday, May 02, 
>> 2008 3:13 PM
>> To: NANOG
>> Subject: [NANOG] Introducing latency for testing?
>>
>> So I want to mimic some latency in a test network for DB replication.
>> I am wondering what other's have used for this? Obviously, the best
>> way to would be to actually have one box across the US or across the
>> globe to actually test against but what if you don't have that?
 >

boxes across the globe have the property of being somewhat less 
deterministic than you'd like if you need repeatability.

> IIRC ipfw can do this using dummynet and the delay directive.

it will also do jitter and drop rate...

to wit, it's exactly what is need here.

there are analogous tools for iptables based platforms.



------------------------------

Message: 7
Date: Sat, 14 Jun 2008 17:18:38 -0400
From: Scott McGrath <mcgrath at fas.harvard.edu>
Subject: Re: DNS problems to RoadRunner - tcp vs udp
To: Jeroen Massar <jeroen at unfix.org>
Cc: nanog at merit.edu
Message-ID: <485435AE.8060609 at fas.harvard.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed


There is no call for insults on this list - Rather thought this list was 
about techincal discussions affecting all of us and keeping DNS alive 
for the majority of our customers certainly qualifies.

We/I am more than aware of the DNS mechanisms and WHY there are there 
trouble is NO DNS server can handle directed TCP attacks even the root 
servers crumbled under directed botnet activity and we have taken the 
decision to accept some collateral damage in order to keep services 
available.     We are a well connected university network with 
multi-gigabit ingress and egress with 10G on Abilene  so we try to 
protect the internet from attacks originating within our borders AND we 
really feel the full wrath of botnets as we do not have a relatively 
slow WAN link to buffer the effects.

Yes - we are blocking TCP too many problems with drone armies and we 
started about a year ago when our DNS servers became unresponsive for no 
apparent reason.   Investigation showed TCP flows of hundreds of 
megabits/sec and connection table overflows from tens of thousands of 
bots all trying to simultaneously do zone transfers and failing tried 
active denial systems and shunning with limited effectiveness.

We are well aware of the host based mechanisms to control zone 
information,  Trouble is with TCP if you can open the connection you can 
DoS so we don't allow the connection to be opened and this is enforced 
at the network level where we can drop at wire speed.     Open to better 
ideas but if you look at the domain in my email address you will see we 
are a target for hostile activity just so someone can 'make their bones'.

Also recall we have a comittment to openess so we would like to make TCP 
services available but until we have effective DNS DoS mitigation which 
can work with 10Gb links It's not going to happen. 

- Scott

Jeroen Massar wrote:
> Scott McGrath wrote:
> [..]
>> For a long time there has been a effective practice of
>>
>> UDP == resolution requests
>> TCP == zone transfers
>
> WRONG. TCP is there as a fallback when the answer of the question is 
> too large. Zone transfer you can limit in your software. If you can't 
> configure your dns servers properly then don't run DNS.
> Also note that botnets have much more effective ways of taking you out.
>
> And sometimes domains actually require TCP because there are too many 
> records for a label eg http://stupid.domain.name/node/651
> If you are thus blocking TCP for DNS resolution you suddenly where 
> blocking google and thus for some people "The Internet".
>
> Also see:
> http://homepages.tesco.net/J.deBoynePollard/FGA/dns-edns0-and-firewalls.html 
>
>
> (Which was the second hit for google(EDNS0) after a link to RFC2671)
>
> Greets,
>  Jeroen
>




------------------------------

Message: 8
Date: Sat, 14 Jun 2008 23:23:44 +0200
From: Simon Leinen <simon.leinen at switch.ch>
Subject: Re: DNS problems to RoadRunner - tcp vs udp
To: Jon.Kibler at aset.com
Cc: nanog at merit.edu
Message-ID: <aaod63g17j.fsf at switch.ch>
Content-Type: text/plain; charset=us-ascii

Jon Kibler writes:
> Also, other than "That's what the RFCs call for," why use TCP for
> data exchange instead of larger UDP packets?

TCP is more robust for large (>Path MTU) data transfers, and less
prone to spoofing.

A few months ago I sent a message to SwiNOG (like NANOG only less
North American and more Swiss) about this topic, trying to explain
some of the tradeoffs:

http://www.mail-archive.com/[email protected]/msg02612.html

Mostly I think that people "approaching this from a security
perspective only" often forget that by fencing in the(ir idea of the)
current status quo, they often prevent beneficial evolution of
protocols as well, contributing to the Internet's "ossification".
-- 
Simon.



------------------------------

Message: 9
Date: Sat, 14 Jun 2008 23:54:49 +0200
From: Jeroen Massar <jeroen at unfix.org>
Subject: Re: DNS problems to RoadRunner - tcp vs udp
To: Scott McGrath <mcgrath at fas.harvard.edu>
Cc: nanog at merit.edu
Message-ID: <48543E29.70809 at spaghetti.zurich.ibm.com>
Content-Type: text/plain; charset="iso-8859-1"

Scott McGrath wrote:
> 
> There is no call for insults on this list

Insults? Where? If you feel insulted by any of the comments made on this 
list by people, then you probably are indeed on the wrong list. But that 
is just me.

> - Rather thought this list was 
> about techincal discussions affecting all of us and keeping DNS alive 
> for the majority of our customers certainly qualifies.

[..blabber about DNS attacks over TCP..]

If I where a botnet herder and I had to take out your site and I was 
going to pick TCP for some magical reason then I would not care about 
your DNS servers, I would just hit your webservers, hard. I mean just 
the 'index.html' (http://www.harvard.edu/) is 24Kb, that is excluding 
pictures and there is bound to be larger data there which you are going 
to send and the bots only have to say "ACK" to once in a while.

Multiply that by say a small botnet of 1M hosts, each just requests that 
24Kb file. You will have a million flows and won't have any way to rate 
limit that or control it. Your link was already full trying to send it 
back to the clients and next to that your server was probably not able 
to process it in the first place. Simple, effective, nothing you can do 
about it, except get way and way more hardware.

If somebody wants to take you out, they will take you out. Just get one 
other box with 10GE (not too hard to do) or just get a million of them 
with a little bit of connectivity (which is quite easy apparently)...

> We/I am more than aware of the DNS mechanisms and WHY there are there 
> trouble is NO DNS server can handle directed TCP attacks even the root 
> servers crumbled under directed botnet activity and we have taken the 
> decision to accept some collateral damage in order to keep services 
> available.

"The root servers crumbled" wow, I must have missed somebody taking out 
all the 13 separate and then individually anycasted root servers. Which 
btw only do UDP as currently '.' is still small enough.

$ dig @a.root-servers.net. . NS +tcp
[..]
;; Query time: 95 msec
;; SERVER: 2001:503:ba3e::2:30#53(2001:503:ba3e::2:30)
;; WHEN: Sat Jun 14 23:45:52 2008
;; MSG SIZE  rcvd: 604

That is only 1 packet to 1 packet, still only 500 bytes. While your 
little webserver would generate 24kb for that same sequence.

>    We are a well connected university network with 
> multi-gigabit ingress and egress with 10G on Abilene  so we try to
> protect the internet from attacks originating within our borders AND we 
> really feel the full wrath of botnets as we do not have a relatively 
> slow WAN link to buffer the effects.

The whole point generally of botnets is just the Denial of Service 
(DoS), if that is because your link is full or the upstreams link is 
full or because the service can't service clients anymore.

But clearly, as you are blocking TCP-DNS you are DoSing yourself 
already, so the botherders win.

Also note that Abilene internally might be 10G and in quite some places 
even 40G, but you still have to hand it off to the rest of the world and 
those will count as those 'slow WAN' links that you think everybody else 
on this planet is behind. (Hint: 10GE is kinda the minimum for most 
reasonably sized ISP's)

> Yes - we are blocking TCP too many problems with drone armies and we 
> started about a year ago when our DNS servers became unresponsive for no 
> apparent reason.   Investigation showed TCP flows of hundreds of 
> megabits/sec and connection table overflows from tens of thousands of 
> bots all trying to simultaneously do zone transfers and failing tried 
> active denial systems and shunning with limited effectiveness.

How is a failed AXFR going to generate a lot of traffic, unless they are 
repeating themselves over and over and over again? Thus effectively just 
packeting you?

Also, are you talking about Recursive or Authoritive DNS servers here?
Where those bots on your network, or where they remote?

> We are well aware of the host based mechanisms to control zone 
> information,  Trouble is with TCP if you can open the connection you can 
> DoS so we don't allow the connection to be opened and this is enforced 
> at the network level where we can drop at wire speed.

Do you mean that the hosts which do TCP are allowed to do transfers or 
not? As in the latter case they can't generate big answers, they just 
get 1 packet back and then end then FIN.

Note also, that if they are simply trying to overload your hosts, UDP is 
much more effective in doing that already and you have that hole wide 
open apparently otherwise you wouldn't have DNS.

> Open to better 
> ideas but if you look at the domain in my email address you will see we 
> are a target for hostile activity just so someone can 'make their bones'.

It probably has nothing to do with the domain name, it more likely has 
something to do with certain services that are available or provided on 
your network.

> Also recall we have a comittment to openess so we would like to make TCP 
> services available but until we have effective DNS DoS mitigation which 
> can work with 10Gb links It's not going to happen.

You think that 10Gb is a 'fat link', amusing ;)

There are various vendors, most likely also reading on this list, who 
can be more than helpful in providing you with all kinds of bad, but 
also a couple of good solutions to most networking issues that you are 
apparently having.

But the biggest issue you seem to have is not knowing what the DoS 
kiddies want to take out and why they want to take it out.

Greets,
  Jeroen

PS: You do know that an "NS" record is not allowed to point to a CNAME I 
hope? (NS3.harvard.edu CNAME ns3.br.harvard.edu. RFC1912 2.4 ;)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 187 bytes
Desc: OpenPGP digital signature
Url : http://mailman.nanog.org/pipermail/nanog/attachments/20080614/29473ad3/attachment.pgp 

------------------------------

_______________________________________________
NANOG mailing list
NANOG at nanog.org
http://mailman.nanog.org/mailman/listinfo/nanog


End of NANOG Digest, Vol 5, Issue 33
************************************


More information about the NANOG mailing list