pontification bloat (was 10GE TOR port buffers (was Re: 10G switch recommendaton))

Sat Jan 28 02:21:05 UTC 2012

In a message written on Sat, Jan 28, 2012 at 11:02:14AM +0900, Randy Bush wrote:
> one problem is that we do not have good tools to look at a link and
> suggest parms.  how did you derive those?

It's actually simple math, it just can get moderate complex.

Let's say you have a 10Mbps ethernet interface, and you want to set
the queue size (in packets).

10Mbps is ~1250000 bytes/sec.

Now, I pick an arbitrary value, this is where experience comes in.
For this example I'm going to say I want no more than 5ms queuing
latency.  5ms/1000sec/ms * 1250000 = 6250 bytes.

I then look at my MTU, we'll go with 1500 here.  6250 / 1500 4.16
packets.

So queueing around 4 full sized packets generates 0-5ms of jitter
on a 10Mbps ethernet, worst case.

How many ms is good?  Well, that depends, a lot.  However I suspect most
people here have seen enough pings they have some idea what is good and
what isn't.

From there you have to look at if there is a hardware ring buffer under
the software QOS (not on most interfaces, but yes on some), and then
look if the buffer is per-VC (subint, whatever) on an interface with
multiple subinterfaces.

This is as much art as science.  My rules of thumb:

- High speed backbone interfaces, 1-3ms of buffer.
- Medium to high speed links inside of a single pop/site, 2-5ms of buffer.
- Low speed access/edge, 5-20ms of buffer.

I have rarely seen an application benefit from more than 20ms of buffer.
Remember, this is per hop.  If you take a 20 hop traceroute and each hop
that 20ms of buffer, you would be waiting 400ms if all the buffers were
full!  That's even if all 20 hops are in the same building, so the RTT
is 1ms.  Imagine how crappy a 1ms RTT would be with random 4/10ths of a
second pauses would be!

Now, here's where it gets non-intuitive.  Reducing the buffers will
_increase_ packet drops, which will make your customers _happier_.
It will also generally smooth out sawtooth patterns (caused by
congestion collapse syncronization, everyone fills the buffer at the
same time, backs off at the same time, etc).  So your links may go from
spiky between 90-100%, to flatlined at 100%, but your customers will be
happier.

Run the math the other way to see how many ms your current buffer size
allows the router to hold.

-- 
       Leo Bicknell - bicknell at ufp.org - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 826 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20120127/416e2e0d/attachment.sig>