MX204 tunnel services BW

Tue Oct 3 06:21:16 UTC 2023

On Mon, 2 Oct 2023 at 20:21, Jeff Behrns via NANOG <nanog at nanog.org> wrote:

> Encountered an issue with an MX204 using all 4x100G ports and a logical
> tunnel to hairpin a VRF.  The tunnel started dropping packets around 8Gbps.
> I bumped up tunnel-services BW from 10G to 100G which made the problem
> worse; the tunnel was now limited to around 1.3Gbps.  To my knowledge with
> Trio PFE you shouldn't have to disable a physical port to allocate bandwidth
> for tunnel-services.  Any helpful info is appreciated.

You might have more luck in j-nsp.

But yes you don't need any physical interface in trio to do tunneling.
I can't explain your problem, and you probably need JTAC help. I would
appreciate it if you'd circle back and tell what the problem was.

How it works is that when PPE decides it needs to tunnel the packet,
you're going to send the packet back to MQ via SERDES (which will then
send it again to some PPE, not the same). I think what that bandwidth
command does is change the stream allocation, you should see it in
'show <MQ/XM...> <#> stream'.

In theory, because PPE can process packet forever (well, until
watchdog kills the PPE for thinking it is stuck) you could very
cheaply do outer+inner at the local PPE, but I think that would mean
that certain features like QoS would not work on the inner interface,
so I think all this expensive recirculation and SERDES consumption is
to satisfy quite limited need, and it should be possible to implement
some 'performance mode' for tunneling, where these MQ/XM provided
features are not available, but performance cost in most cases is
negligible.

In parallel to opening the JTAC case, you might want to try to
experiment in which FPC/PIC you set the tunneling bandwidth to. I
don't understand how the tunneling would work if the MQ/XM is remote,
like would you then also steal fabric capacity every time you tunnel,
not just  MQ>LU>MQ>LU SERDES, but MQ>LU>MQ>FAB>MQ>LU. So intuitively I
would recommend ensuring you have the bandwidth configured at the
local PFE, if you don't know what the local PFE is, just configure it
everywhere?
Also you could consult several counters to see if some stream or
fabric is congested, and these tunneled packets are being sent over
congested fabric every time with lower fabric qos.

I don't understand why the bandwidth command is a thing, and why you
can configure where it is. To me it seems obvious they should always
handle tunneling strictly locally, never over fabric, because you
always end up stealing more capacity if you send it to remote MQ. That
is, implicitly it should be on for every MQ, and every PPE tunnel via
local MQ.

-- 
  ++ytti