TCP time_wait and port exhaustion for servers

William Herrin bill at
Wed Dec 5 20:56:30 UTC 2012

On Wed, Dec 5, 2012 at 2:55 PM, Ray Soucy <rps at> wrote:
> For each second that goes by you remove X addresses from the available
> pool of ports for new connections for whatever the TCP_TIMEWAIT_LEN is
> set to (60 seconds by default in Linux).
> In this case it's making quick connections for HTTP requests (most of
> which finish in less than a second).
> Say you have a pool of 30,000 ports and 500 new connections per second
> (typical):
> 1 second goes by you now have 29500
> 10 seconds go by you now have 25000
> 30 seconds go by you now have 15000
> at 59 seconds you get to 29500,
> at 60 you get back 500 and stay at 29500 and that keeps rolling at
> 29500.  Everyone is happy.

The thing is, Linux doesn't behave quite that way.

If you do an anonymous connect(), that is you socket() and then
connect() without a bind() in the middle, then the limit applies *per
destination IP:port pair*. So, you should be able to do 30,000
connections to port 80, another 30,000 connections to port 80, and so on.

You should only fail if you A) bump against the top of NR_OPEN or B)
try to do a massive number of TCP connections to the same remote IP

Try it: set up a listener on discard that just closes the connection
and repeat connect() to until you get an error. Then confirm
that you're out of ports:

telnet 9
telnet: Unable to connect to remote host: Cannot assign requested address

And confirm that you can still make outbound connections to a
different IP address:

telnet 9
Connected to
Escape character is '^]'.
Connection closed by foreign host.

Bill Herrin

William D. Herrin ................ herrin at  bill at
3005 Crane Dr. ...................... Web: <>
Falls Church, VA 22042-3004

More information about the NANOG mailing list