<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-CA link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='mso-fareast-language:EN-US'>If you still need netflow to gain some visibility on what’s happening, you could check the percentage of netflow export. <o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Usually 1/1000 is good or 0.1%. Maybe for you 1/1 000 000 could be good enough too.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>If 100% was used, then indeed there are some real time performance penalties. Not much people need an accurate 100% of netflow exports. If you need 100% accuracy, then you need dedicated hardware.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>0% or totally disabled is also often very good enough if you don’t need visibility. </span><span style='font-family:"Segoe UI Emoji",sans-serif;mso-fareast-language:EN-US'>😊</span><span style='mso-fareast-language:EN-US'><o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Netflow is useful in my opinion, but maybe not for every case.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Jean<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US>From:</span></b><span lang=EN-US> NANOG <nanog-bounces+jean=ddostest.me@nanog.org> <b>On Behalf Of </b>Colin Legendre<br><b>Sent:</b> December 9, 2021 7:18 PM<br><b>To:</b> Brian Turnbow <b.turnbow@twt.it><br><b>Cc:</b> nanog <nanog@nanog.org><br><b>Subject:</b> Re: Latency/Packet Loss on ASR1006<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>NBAR was not enabled.. just netflow export.. and that was enough..<br clear=all><o:p></o:p></p><div><div><div><div><div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>---<br>Colin Legendre<br>President and CTO<br><br>Coextro - Unlimited. Fast. Reliable.<br>w: <a href="http://www.coextro.com" target="_blank">www.coextro.com</a><br>e: <a href="mailto:clegendre@coextro.com" target="_blank">clegendre@coextro.com</a><br><br>p: 647-693-7686 ext.101<br>m: 416-560-8502<br>f: 647-812-4132<o:p></o:p></p></div></div></div></div></div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Thu, Dec 9, 2021 at 7:17 PM Colin Legendre <<a href="mailto:clegendre@coextro.com">clegendre@coextro.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><p class=MsoNormal>Thanks for this.. turned off netflow export.. and it dropped our qfp load from 44% to 18%. ugh..<br clear=all><o:p></o:p></p><div><div><div><div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'>---<br>Colin Legendre<o:p></o:p></p></div></div></div></div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Thu, Dec 9, 2021 at 4:22 AM Brian Turnbow via NANOG <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><p class=MsoNormal style='margin-bottom:12.0pt'><br><br>> On 11/26/2021 1:09 PM, Colin Legendre wrote:<br>> > Hi,<br>> ><br>> > We have ...<br>> ><br>> > ASR1006 that has following cards...<br>> > 1 x ESP40<br>> > 1 x SIP40<br>> > 4 x SPA-1x10GE-L-V2<br>> > 1 x 6TGE<br>> > 1 x RP2<br>> ><br>> > We've been having latency and packet loss during peak periods...<br>> ><br>> > We notice all is good until we reach 50% utilization on output of...<br>> ><br>> > 'show platform hardware qfp active datapath utilization summary'<br>> ><br>> > Literally ... 47% good... 48% good... 49% latency to next hop goes<br>> > from 1ms to 15-20ms... 50% we see 1-2% packet-loss and 30-40ms<br>> > latency... 53% we see 60-70ms latency and 8-10% packet loss.<br>> ><br>> > Is this expected... the ESP40 can only really push 20G and then starts<br>> > to have performance issues?<br>> ><br><br>He had a similar issue about 4 years ago.<br>We were showing packet loss and drops getting progressively worse and the router was falling over when reaching about 70% of usage.<br>We could see the interface reliability go down and input errors due to overruns on the interfaces.<br>Cisco blamed it on microburtst not being able to be handled under load.<br><br><br>"We were able to replicate this scenario in our lab as well.<br>QFP under high load generated input errors and overruns which in turn led to unicast failures/ drops/ latency.<br>The issue is not consistent with QFP % utilization as sometimes with even 80%+ traffic, we do not see the drops:"<br><br>And recommended removing traffic or upgrading esp.<br><br>One of our guys disabled nbar on the router and the problem disappeared.<br>I would suggest taking a look at what features you are using and if you can try and disable them to see if it makes any impact.<br>We then upgraded esps and all has been fine since.<br><br>Brian<o:p></o:p></p></blockquote></div></blockquote></div></div></body></html>