Extreme congestion (was Re: inter-domain link recovery)

michael.dillon at bt.com michael.dillon at bt.com
Thu Aug 16 22:20:19 UTC 2007

> The TCPs don't slow down. They use the bandwidth you have 
> made available instead.
> in your words, "the traffic on the new circuit is suddenly 
> greater than 100% of the old one".


To be honest, I first encountered this when Avi Freedman upgraded one of
his upstream connections from T1 to DS3 and either Avi, or one of his
employees mentioned this on inet-access or nanog. So I did a bit of
digging and discovered that other people had noticed that TCP traffic
tends to be fractal (or multi-fractal) in nature. That means that the
peaks which cause this effect are hard to get rid of entirely.

> To his surprise, with the standing load > 95% and 
> experiencing 20% loss at 311 MBPS, doubling the rate to 622 
> MBPS resulted in links with a standing load > 90% and 4% 
> loss. He still needed more bandwidth. After we walked 
> offstage, I explained TCP to him...

That is something that an awful lot of operations and capacity planning
people do not understand. They still think in terms of pipes with TCP
flavoured water flowing in them. But this is exactly the behavior that
you would expect from fractal traffic. The doubled capacity gave enough
headroom for some of the peaks to get through, but not enough for all of
them. On Ebone in Europe we used to have 40% as our threshold for
upgrading core circuits. 

> I'm pushing an agenda in the open source world to add  
> some concept of locality, with the purpose of moving traffic off ISP  
> networks when I can. I think the user will be just as happy or  
> happier, and folks pushing large optics will certainly be.

When you hear stories like the Icelandic ISP who discovered that P2P was
80% of their submarine bandwidth and promptly implemented P2P
throttling, I think that the open source P2P will be driven to it by
their user demand. 

--Michael Dillon

More information about the NANOG mailing list