TCP time_wait and port exhaustion for servers

Thu Dec 6 03:29:24 UTC 2012

In message <20121206004909.B302F2CA212F at drugs.dv.isc.org>, Mark Andrews writes:
> 
> In message <201212052325.qB5NPrZe005631 at xs8.xs4all.nl>, "Miquel van Smoorenburg"
>  writes:
> > In article <xs4all.20121205220127.7F6F12CA0F17 at drugs.dv.isc.org> you write:
> > >
> > >In message <CAP-guGW6oXo=UfTfg+SDiFjB4=qxPShO+YfK6vxnLkCC58PvgQ at mail.gmail.co
> > m>,
> > > William Herrin writes:
> > >> The thing is, Linux doesn't behave quite that way.
> > >> 
> > >> If you do an anonymous connect(), that is you socket() and then
> > >> connect() without a bind() in the middle, then the limit applies *per
> > >> destination IP:port pair*. So, you should be able to do 30,000
> > >> connections to 192.168.1.1 port 80, another 30,000 connections to
> > >> 192.168.1.2 port 80, and so on.
> > >
> > >The socket api is missing a bind + connect call which restricts the
> > >source address when making the connect.  This is needed when you
> > >are required to use a fixed source address.
> > 
> > William was talking about the destination address. Linux (and I would
> > hope any other network stack) can really open a million connections
> > from one source address, as long as it's not to one destination address
> > but to lots of different ones. It's not the (srcip,srcport) tuple that
> > needs to be unique; it's the (srcip,srcport,dstip,dstport) tuple.
> > 
> > Anyway, you can actually bind to a source address and still have a
> > dynamic source port; just use port 0. Lots of tools do this.
> > 
> > (for example, strace nc -s 127.0.0.2 127.0.0.1 22 and see what it does)
> > 
> > Mike.
> 
> Eventually the bind call fails.  Below was a 
> 
> counter: dest address in hex
> 
> 16376: 1a003ff9
> 16377: 1a003ffa
> bind: before bind: Can't assign requested address
> 16378: 1a003ffb
> connect: Can't assign requested address
> bind: before bind: Can't assign requested address
> 
> and if you remove the bind() the connect fails
> 
> 16378: 1a003ffb
> 16379: 1a003ffc
> connect: Can't assign requested address
> 16380: 1a003ffd
> 
> this is with a simple loop
> 
> 	socket()
> 	ioctl(FIONBIO)
> 	bind(addr++:80)
> 	connect()
> 
> I had a firewall dropping the connection attempts

To get more one needs to setsockopt SO_REUSEADDR but that consumes
all the port space so applications that need to listen for incoming
connections on the same machine will break.

If you also set IP_PORTRANGE_HIGH and have configured the system
so that the high range does not match the default range then you
can avoid the above issue.  This is the default configuration for
some but not all platforms.

Sockets with IP_PORTRANGE_HIGH set are not expected to accept
incoming traffic.

So if you are making a out bound connections you should set
SO_REUSEADDR and IP_PORTRANGE_HIGH options on the socket to avoid
local port limits.  Not most applications do not do this which is
fine until you are using 10's of thousands of outgoing sockets.

Mark
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: marka at isc.org