Tail Drops and TCP Slow Start
Jon 'tex' Boone
tex at delamancha.org
Tue Dec 11 09:19:51 UTC 2001
"Murphy, Brennan" <Brennan_Murphy at NAI.com> writes:
> If I have a DS3 or OC3 handling mounds and mounds of FTP download
> traffic, what is the easiest way to detect if the bandwidth in use is
> falling into a classic Tail Drop pattern? According to a Cisco book
> I am reading, the bandwidth utilization should graph in a "sawtooth"
> pattern of gradual increases in accordance with multiple machines
> gradually increasing via TCP slow start and then sharp drops. Will
> this only happen when the utilization approaches 100%. (maybe dumb
My rule #1 of troubleshooting performance problems is that most of
the effect that you are seeing is due to a single problem somewhere
along the path. Fixing that 1 item will yield a huge boost in
performance. Fixing the rest of the problems will yield smaller,
incremental boosts - although possibly still substantial.
If the sawtooth in the graph represents a single session going into
error recover (i.e. slow-start), then you *might* see this without
necessarily seeing overwhelming evidence of it on the Cisco. If the
sawtooth represents multiple sessions entering slow start nearly
simultaneously (aka global synchronization), then it would be much
easier to capture evidence of this via "show interface" stats.
The bursty nature of data traffic is such that you can experience
temporary congestion events that are "smoothed over" by queueing on
the outbound interface. If these events are brief in duration and
separated sufficiently in time, you will not necessarily see any
indication of it via the "show interfaces" output. The shorter you
have set your load-interval, the more likely you are to see the
bandwidth increase represented by the burst (although 30 seconds is
long enough to hide bursts on lightly loaded links).
Note: it is difficult to get instantaneous measures of queueing
reliably from a Cisco. Best is to check the output of "show
controller cbus" (on 75xx series) which will show you the hardware
details: look for the txacc and txlimit values. txlimit is the
maximum # of items allowed in the transmit ring for that interface,
while txacc == (txlimit - # items in the queue). This command needs
to be issued repeatedly (and quickly) to get some sense of the queue
size. This is *slightly* better than looking at the output of show
interface, but still leaves a lot to be desired. :-)
Once your tx queue fills up completely (txacc == 0), all packets
scheduled for transmission on that interface are dropped. As your
bandwidth utilization approaches 100%, you will see more and more
queueing take place, meaning that the chances of the next incoming
packet being dropped increase significantly as the load goes up. I
doubt that you would see global synchronization until the load on your
link was very near to 100%, but I haven't done the traffic studies to
prove it. :-)
RED attempts to prevent this situation by pro-actively dropping
packets from the tx queue before it fills up. Using an
exponentially-weighted average (to smooth out burstiness) of queue
size to determine how likely it is that the current packet will be
dropped, RED will tend to hit the "high bandwidth" users first,
leaving the smaller users relatively unharmed. If RED does its job
correctly, then you will not see global synchronization, although a
graph of the throughput of a single FTP session that happened to be
policed by RED would demonstrate the sawtooth.
> Should I be able to do a show buffers and see misses or is there
> some better way to detect other than via graphing?
You are very unlikely to see this via "show buffers", as this is not
likely to be caused by your device running out of memory if your cards
are sized correctly. The only way to tell is by looking at the
instantaneous measures of queue size if you are looking for a single
session performance drop (in the face of near constant high background
utilization). If you are seeing global synchronization, then you
should see a *big* dip in your usage via "show interface" when set to
30 second load-interval.
> Anyone have experience with WRED to handle ftp congestion?
RED is specifically designed to deal with this problem. WRED and
dWRED do a decent job, but nothing can help you if you simply have
more aggregate demand for bandwidth than your interface can support.
And, neither WRED nor dWRED work for UDP applications/DoS attacks.
Jon Allen Boone
tex at delamancha.org
More information about the NANOG