IPv4 Address Exhaustion Effects on the Earth

Wed Apr 6 22:43:30 UTC 2011

On 04/05/2011 06:17 PM, Jared Mauch wrote:
> On Apr 5, 2011, at 6:07 PM, Jim Gettys wrote:
>
>> On 04/05/2011 05:59 PM, Michael Proto wrote:
>>> On Tue, Apr 5, 2011 at 5:38 PM, Jared Mauch<jared at puck.nether.net>   wrote:
>>>> On Apr 4, 2011, at 4:30 PM, Jim Gettys wrote:
>>>>
>>>>> Note that the paper "Characterizing Residential Broadband Networks" by Dischinger, et. al. indicates that a large fraction (in their 2 year old sample, 30% or so) of broadband head ends are running without RED, and should be doing so if at all possible; alternatives are years out by the time they are tested and deployed, and operators running without it in congested systems are inflicting pain on their customers.
>>>> Something I've observed is if you are sending data 'upstream' on the cable modem setup i have (16 down/ 2 up) and you saturate the upstream, the buffering destroys any downstream capability at the same time.  I'm not even sure where to start diagnosing to explaining this to the carrier involved, as this isn't the desired behavior of a "business class" service.
>>>>
>>>> - Jared
>>>>
>>> Isn't this just a case or prioritizing outbound ACKs?
>>>
>>> http://www.benzedrine.cx/ackpri.html
>>>
>> Nope.  Your acks get delayed to what you are sending upstream, behind the downstream traffic.
>>
>> Bufferbloat hurts both directions, once saturation occurs and your latencies start to go up.
>>
>> Note that on many of these links, the RTT becomes (literally) as though you are half way (or further than) the moon.
>>
> I sent a private reply, but I guess i'll post some of it here:
>
> 1) there are no ways to identify the devices doing the buffering and/or drop counts

This isn't really true: you basically do "ping" combined with 
"traceroute" while saturating a path.  The hop where there is unexpected 
additional latency not present when the you don't saturate the path is 
the culprit.

You can't easily figure out where inside a bottleneck the buffering is, 
unless you have some way to monitor or control the buffering inside it 
(e.g. twist the transmit queue or transmit rings in Linux, as an example).

> 2) I can obviously feed the cable modem much faster on the lan vs what it can send upstream
>
> Doing things like rate-limiting/QoS are merely just papering over the problem.

Papering over the problem isn't all bad, if it allows you to hide 
egregious size buffers in a bottleneck link over which you'd otherwise 
have no control.

It allows me to take my latency/jitter on my home cable service from 
about 400ms down to 10ms (at some cost in bandwidth).  This means I 
actually can have voip or other latency sensitive applications work (so 
long as I ensure that the broadband link is the bottleneck; if you home 
wireless network's effective bandwidth drops below that of your 
broadband, then the bottleneck becomes the 802.11 hop, and you'll see 
queues in your host and home router, rather than in the broadband hop.  
Notice that this means with symmetric 25/25 FIOS service, you get 
bufferbloat in your host and wireless router, as I did (actual data 
transfer rates of 802.11g is only about 20Mbps, not the 54 signalling 
rate marketed).

> I would take a T1 and rate-limit it to 1.2Mb/s for TCP to allow VoIP to work.  Junipers can buffer up to 1 second on these low-speed interfaces, which obviously creates the problems you describe.

Bufferbloat is (nearly) everywhere.

You'd better see if you can run RED or some AQM.  The issues with RED 
revolve around the fact that it is a flawed algorithm that requires 
proper tuning to be effective, and it can't handle situations such as 
wireless or broadband with time variable behaviour such as Comcast's 
Powerboost.

It is exactly the fact that RED requires tuning that means that it is 
often not enabled when it should be: but it is often the only tool we've 
got right now.

> There are a lot more problems with the gateway devices, such as the forcible dns proxy that exists.
>
Right now, the home router market is a cesspool of scum and villainy. 
Our best hope is the rebel alliance; I think we'll get OpenWRT 
straightened out long  before the commercial vendors do.
                 - Jim