10GE TOR port buffers (was Re: 10G switch recommendaton)

Fri Jan 27 23:03:49 UTC 2012

In a message written on Fri, Jan 27, 2012 at 11:30:14PM +0100, bas wrote:
> While your reasoning holds truth it does not explain why the expensive
> chassis solution (good) makes my customers happy, and the cheaper TOR
> solution makes my customers unhappy.....
> 
> Bufferbloat does not matter to them as jitter and latency does not matter.
> As long as the TCP window size negotioation is not reset the total
> amount of bit/sec increases for them.

I obviously don't know your application.  The bufferbloat problem
exists for 99.99% of the standard applications in the world.  There
are, however, a few corner cases.  For instance, if you want to
move a _single_ TCP stream at more than 1Gbps you need deep buffers.
Dropping a single packet slows throughput too much due to a slow-start
event.  For most of the world with hundreds or thousands of TCP
streams across a single port, such problems never occur.

> If deep buffers are bad I would expect high-end chassis solutions not
> to offer them either.
> But the market seems to offer expensive deep buffer chassis solutions
> and cheap (per 10GE) TOR solutions.

The margin on a top-of-rack switch is very low.  48 port gige with
10GE uplinks are basically commodity boxes, with plenty of competition.
Saving $100 on the bill of materials by cutting out some buffer
makes the box more competitive when it's at a $2k price point.

In contrast, large, modular chasses have a much higher margin.  They are
designed with great flexability, to take things like firewall modules
and SSL accelerator cards.  There are configs where you want some (not
much) buffer due to these active appliances in the chassis, plus it is
easier to hide an extra $100 of RAM in a $100k box.

Also, as was pointed out to me privately, it is also important to loook
at adaptive queue management features.  The most famous is WRED, but
there are other choices.  Having a queue management solution on your
routers and switches that works in concert with the congestion control
mechanism used by the end stations always results in better goodput.
Many of the low end switches have limited or no AQM choices, while the
higher end switches with fancier ASICs can default to something like
WRED.  Be sure it is the deeper buffers that are making the difference,
and not simply some queue management.

-- 
       Leo Bicknell - bicknell at ufp.org - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 826 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20120127/ed6a5da6/attachment.sig>