TCP time_wait and port exhaustion for servers
marka at isc.org
Wed Dec 5 22:01:27 UTC 2012
In message <CAP-guGW6oXo=UfTfg+SDiFjB4=qxPShO+YfK6vxnLkCC58PvgQ at mail.gmail.com>,
William Herrin writes:
> On Wed, Dec 5, 2012 at 2:55 PM, Ray Soucy <rps at maine.edu> wrote:
> > For each second that goes by you remove X addresses from the available
> > pool of ports for new connections for whatever the TCP_TIMEWAIT_LEN is
> > set to (60 seconds by default in Linux).
> > In this case it's making quick connections for HTTP requests (most of
> > which finish in less than a second).
> > Say you have a pool of 30,000 ports and 500 new connections per second
> > (typical):
> > 1 second goes by you now have 29500
> > 10 seconds go by you now have 25000
> > 30 seconds go by you now have 15000
> > at 59 seconds you get to 29500,
> > at 60 you get back 500 and stay at 29500 and that keeps rolling at
> > 29500. Everyone is happy.
> The thing is, Linux doesn't behave quite that way.
> If you do an anonymous connect(), that is you socket() and then
> connect() without a bind() in the middle, then the limit applies *per
> destination IP:port pair*. So, you should be able to do 30,000
> connections to 192.168.1.1 port 80, another 30,000 connections to
> 192.168.1.2 port 80, and so on.
The socket api is missing a bind + connect call which restricts the
source address when making the connect. This is needed when you
are required to use a fixed source address.
> You should only fail if you A) bump against the top of NR_OPEN or B)
> try to do a massive number of TCP connections to the same remote IP
> Try it: set up a listener on discard that just closes the connection
> and repeat connect() to 127.0.0.5 until you get an error. Then confirm
> that you're out of ports:
> telnet 127.0.0.5 9
> Trying 127.0.0.5...
> telnet: Unable to connect to remote host: Cannot assign requested address
> And confirm that you can still make outbound connections to a
> different IP address:
> telnet 127.0.0.4 9
> Trying 127.0.0.4...
> Connected to 127.0.0.4.
> Escape character is '^]'.
> Connection closed by foreign host.
> Bill Herrin
> William D. Herrin ................ herrin at dirtside.com bill at herrin.us
> 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
> Falls Church, VA 22042-3004
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: marka at isc.org
More information about the NANOG