TCP time_wait and port exhaustion for servers

William Herrin bill at herrin.us
Wed Dec 5 20:56:30 UTC 2012


On Wed, Dec 5, 2012 at 2:55 PM, Ray Soucy <rps at maine.edu> wrote:
> For each second that goes by you remove X addresses from the available
> pool of ports for new connections for whatever the TCP_TIMEWAIT_LEN is
> set to (60 seconds by default in Linux).
>
> In this case it's making quick connections for HTTP requests (most of
> which finish in less than a second).
>
> Say you have a pool of 30,000 ports and 500 new connections per second
> (typical):
> 1 second goes by you now have 29500
> 10 seconds go by you now have 25000
> 30 seconds go by you now have 15000
> at 59 seconds you get to 29500,
> at 60 you get back 500 and stay at 29500 and that keeps rolling at
> 29500.  Everyone is happy.


The thing is, Linux doesn't behave quite that way.

If you do an anonymous connect(), that is you socket() and then
connect() without a bind() in the middle, then the limit applies *per
destination IP:port pair*. So, you should be able to do 30,000
connections to 192.168.1.1 port 80, another 30,000 connections to
192.168.1.2 port 80, and so on.

You should only fail if you A) bump against the top of NR_OPEN or B)
try to do a massive number of TCP connections to the same remote IP
address.

Try it: set up a listener on discard that just closes the connection
and repeat connect() to 127.0.0.5 until you get an error. Then confirm
that you're out of ports:

telnet 127.0.0.5 9
Trying 127.0.0.5...
telnet: Unable to connect to remote host: Cannot assign requested address

And confirm that you can still make outbound connections to a
different IP address:

telnet 127.0.0.4 9
Trying 127.0.0.4...
Connected to 127.0.0.4.
Escape character is '^]'.
Connection closed by foreign host.

Regards,
Bill Herrin


-- 
William D. Herrin ................ herrin at dirtside.com  bill at herrin.us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004




More information about the NANOG mailing list