TCP time_wait and port exhaustion for servers

Sat Dec 8 01:08:11 UTC 2012

+1

Thanks for the tip, this looks very useful.

Looks like it was only introduced in 2.6.35, we're still on 2.6.32 ...
might be worth the upgrade, it just takes so long to test new kernel
versions in this application.

We ended up dropping TCP_TIMEWAIT_LEN to 30 seconds as a band-aid for
now, along with the expanded port range.
In talking to others 20 seconds seems to be 99%+ safe, with the sweet
spot seeming to be 24 seconds or so.  So we opted to just go with 30
seconds and be cautious, even though others claim going as low as 10
or 5 seconds without issue.  I'll let people know if it introduces any
problems.

In talking with the author of HAproxy, he seems to be in the camp that
using SO_LINGER of 0 might be the way to go, but is unsure of how
servers would respond to it; we'll likely try a build with that method
and see what happens at some point.

On Fri, Dec 7, 2012 at 4:51 PM, Matthew Palmer <mpalmer at hezmatt.org> wrote:
> On Thu, Dec 06, 2012 at 08:58:10AM -0500, Ray Soucy wrote:
>> > net.ipv4.tcp_keepalive_intvl = 15
>> > net.ipv4.tcp_keepalive_probes = 3
>> > net.ipv4.tcp_keepalive_time = 90
>> > net.ipv4.tcp_fin_timeout = 30
>>
>> As discussed, those do not affect TCP_TIMEWAIT_LEN.
>>
>> There is a lot of misinformation out there on this subject so please
>> don't just Google for 5 min. and chime in with a "solution" that you
>> haven't verified yourself.
>>
>> We can expand the ephemeral port range to be a full 60K (and we have
>> as a band-aid), but that only delays the issue as use grows.  I can
>> verify that changing it via:
>>
>> echo 1025 65535 > /proc/sys/net/ipv4/ip_local_port_range
>>
>> Does work for the full range, as a spot check shows ports as low as
>> 2000 and as high as 64000 being used.
>
> I can attest to the effectiveness of this method, however be sure and add
> any ports in that range that you use as incoming ports for services to
> /proc/sys/net/ipv4/ip_local_reserved_ports, otherwise the first time you
> restart a service that uses a high port (*cough*NRPE*cough*), its port will
> probably get snarfed for an outgoing connection and then you're in a sad,
> sad place.
>
> - Matt
>
> --
> [An ad for Microsoft] uses the musical theme of the "Confutatis Maledictis"
> from Mozart's Requiem. "Where do you want to go today?" is on the screen,
> while the chorus sings "Confutatis maledictis, flammis acribus addictis,".
> Translation: "The damned and accursed are convicted to the flames of hell."
>
>

-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net