TCP time_wait and port exhaustion for servers

Wed Dec 5 22:01:27 UTC 2012

In message <CAP-guGW6oXo=UfTfg+SDiFjB4=qxPShO+YfK6vxnLkCC58PvgQ at mail.gmail.com>,
 William Herrin writes:
> On Wed, Dec 5, 2012 at 2:55 PM, Ray Soucy <rps at maine.edu> wrote:
> > For each second that goes by you remove X addresses from the available
> > pool of ports for new connections for whatever the TCP_TIMEWAIT_LEN is
> > set to (60 seconds by default in Linux).
> >
> > In this case it's making quick connections for HTTP requests (most of
> > which finish in less than a second).
> >
> > Say you have a pool of 30,000 ports and 500 new connections per second
> > (typical):
> > 1 second goes by you now have 29500
> > 10 seconds go by you now have 25000
> > 30 seconds go by you now have 15000
> > at 59 seconds you get to 29500,
> > at 60 you get back 500 and stay at 29500 and that keeps rolling at
> > 29500.  Everyone is happy.
> 
> 
> The thing is, Linux doesn't behave quite that way.
> 
> If you do an anonymous connect(), that is you socket() and then
> connect() without a bind() in the middle, then the limit applies *per
> destination IP:port pair*. So, you should be able to do 30,000
> connections to 192.168.1.1 port 80, another 30,000 connections to
> 192.168.1.2 port 80, and so on.

The socket api is missing a bind + connect call which restricts the
source address when making the connect.  This is needed when you
are required to use a fixed source address.

> You should only fail if you A) bump against the top of NR_OPEN or B)
> try to do a massive number of TCP connections to the same remote IP
> address.
> 
> Try it: set up a listener on discard that just closes the connection
> and repeat connect() to 127.0.0.5 until you get an error. Then confirm
> that you're out of ports:
> 
> telnet 127.0.0.5 9
> Trying 127.0.0.5...
> telnet: Unable to connect to remote host: Cannot assign requested address
> 
> And confirm that you can still make outbound connections to a
> different IP address:
> 
> telnet 127.0.0.4 9
> Trying 127.0.0.4...
> Connected to 127.0.0.4.
> Escape character is '^]'.
> Connection closed by foreign host.
> 
> Regards,
> Bill Herrin
> 
> 
> -- 
> William D. Herrin ................ herrin at dirtside.com  bill at herrin.us
> 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
> Falls Church, VA 22042-3004
> 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: marka at isc.org