<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">The reason why we need larger buffers on some applications is because of TCP implementation detail. When TCP window grows in size (it grows exponentially) the newly created window size is bursted on to the wire at sender speed.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">If sender is significantly higher speed than receiver, someone needs to store these bytes, while they are serialised at receiver speed. If we cannot store them, then the window cannot grow to accommodate the banwdith*delay product and the receiver cannot observe ideal TCP receive rate. </div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">If we'd change TCP sender to bandwidth estimation, and newly created window space would be serialised at estimated receiver rate then we would need dramatically less buffers. However this less aggressive TCP algorithm would be outcompeted by new reno reducing bandwidth estimation to approach zero.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Luckily almost all traffic is handled by few players, if they agree to change to well behaved TCP (or QUIC) algorithm, it doesn't matter much if the long tail is badly behaving TCP. </div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 9 Apr 2021 at 17:13, Mike Hammett <<a href="mailto:nanog@ics-il.net">nanog@ics-il.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:rgb(0,0,0)">I have seen the opposite, where small buffers impacted throughput.<br><br>Then again, it was observation only, no research into why, other than superficial.<div><br><div><span name="x"></span><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">-----</span><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">Mike Hammett</span><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.ics-il.com/" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank">Intelligent Computing Solutions</a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/ICSIL" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/fbicon.png" style="border: 0pt none;"></a><a href="https://plus.google.com/+IntelligentComputingSolutionsDeKalb" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/googleicon.png" style="border: 0pt none;"></a><a href="https://www.linkedin.com/company/intelligent-computing-solutions" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/linkedinicon.png" style="border: 0pt none;"></a><a href="https://twitter.com/ICSIL" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/twittericon.png" style="border: 0pt none;"></a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.midwest-ix.com/" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank">Midwest Internet Exchange</a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/mdwestix" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/fbicon.png" style="border: 0pt none;"></a><a href="https://www.linkedin.com/company/midwest-internet-exchange" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/linkedinicon.png" style="border: 0pt none;"></a><a href="https://twitter.com/mdwestix" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/twittericon.png" style="border: 0pt none;"></a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.thebrotherswisp.com/" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank">The Brothers WISP</a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/thebrotherswisp" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/fbicon.png" style="border: 0pt none;"></a><a href="https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img src="http://www.ics-il.com/images/youtubeicon.png" style="border: 0pt none;"></a><span name="x"></span><br></div><hr id="gmail-m_5575296919609321463zwchr"><div style="color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Tom Beecher" <beecher@beecher.cc><br><b>To: </b>"Mike Hammett" <<a href="mailto:nanog@ics-il.net" target="_blank">nanog@ics-il.net</a>><br><b>Cc: </b>"Dmitry Sherman" <<a href="mailto:dmitry@interhost.net" target="_blank">dmitry@interhost.net</a>>, "NANOG" <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>><br><b>Sent: </b>Friday, April 9, 2021 8:40:00 AM<br><b>Subject: </b>Re: Trident3 vs Jericho2<br><br><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:13.3333px">If you have all the same port speed, small buffers are fine. If you have 100G and 1G ports, you'll need big buffers wherever the transition to the smaller port speed is located.</span></blockquote><div><br></div><div>While the larger buffer there you are likely to be severely impacting application throughput.</div><div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 9, 2021 at 9:05 AM Mike Hammett <<a href="mailto:nanog@ics-il.net" target="_blank">nanog@ics-il.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:rgb(0,0,0)">What I've observed is that it's better to have a big buffer device when you're mixing port speeds. The more dramatic the port speed differences (and the more of them), the more buffer you need.<div><br></div><div>If you have all the same port speed, small buffers are fine. If you have 100G and 1G ports, you'll need big buffers wherever the transition to the smaller port speed is located.</div><div><br><div><span></span><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">-----</span><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline">Mike Hammett</span><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.ics-il.com/" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank">Intelligent Computing Solutions</a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/ICSIL" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/fbicon.png"></a><a href="https://plus.google.com/+IntelligentComputingSolutionsDeKalb" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/googleicon.png"></a><a href="https://www.linkedin.com/company/intelligent-computing-solutions" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/linkedinicon.png"></a><a href="https://twitter.com/ICSIL" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/twittericon.png"></a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.midwest-ix.com/" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank">Midwest Internet Exchange</a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/mdwestix" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/fbicon.png"></a><a href="https://www.linkedin.com/company/midwest-internet-exchange" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/linkedinicon.png"></a><a href="https://twitter.com/mdwestix" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/twittericon.png"></a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="http://www.thebrotherswisp.com/" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank">The Brothers WISP</a><br style="color:rgb(0,0,0);font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://www.facebook.com/thebrotherswisp" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/fbicon.png"></a><a href="https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg" style="font-family:"times new roman";font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" rel="nofollow noopener noreferrer" target="_blank"><img style="border: 0pt none;" src="http://www.ics-il.com/images/youtubeicon.png"></a><span></span><br></div><hr id="gmail-m_5575296919609321463gmail-m_-7717366008892160544zwchr"><div style="color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Dmitry Sherman" <<a href="mailto:dmitry@interhost.net" target="_blank">dmitry@interhost.net</a>><br><b>To: </b><a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a><br><b>Sent: </b>Friday, April 9, 2021 7:57:05 AM<br><b>Subject: </b>Trident3 vs Jericho2<br><br>Once again, which is better shared buffer featurerich or fat buffer switches?<br>When its better to put big buffer switch? When its better to drop and retransmit instead of queueing?<br><br>Thanks.<br>Dmitry<br></div><br></div></div></div></blockquote></div>
</div><br></div></div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">  ++ytti</div>