MX204 tunnel services BW

Tue Oct 3 11:11:49 UTC 2023

You can configure tunnel bandwidth everywhere, but you can’t configure
a given tunnel everywhere, you have to assign it to a particular FPC/PIC/0.

For example, with:
set chassis fps 2 pic 3 tunnel-services bandwidth 10g

You need to create gr-2/3/0 interfaces for tunnels to use that PFE.

You can create multiple tunnel-services bandwidth entries on multiple
PICs, but you can only put a given tunnel on one gr-x/y/0 interface.

Owen

> On Oct 2, 2023, at 23:21, Saku Ytti <saku at ytti.fi> wrote:
> 
> On Mon, 2 Oct 2023 at 20:21, Jeff Behrns via NANOG <nanog at nanog.org> wrote:
> 
>> Encountered an issue with an MX204 using all 4x100G ports and a logical
>> tunnel to hairpin a VRF.  The tunnel started dropping packets around 8Gbps.
>> I bumped up tunnel-services BW from 10G to 100G which made the problem
>> worse; the tunnel was now limited to around 1.3Gbps.  To my knowledge with
>> Trio PFE you shouldn't have to disable a physical port to allocate bandwidth
>> for tunnel-services.  Any helpful info is appreciated.
> 
> You might have more luck in j-nsp.
> 
> But yes you don't need any physical interface in trio to do tunneling.
> I can't explain your problem, and you probably need JTAC help. I would
> appreciate it if you'd circle back and tell what the problem was.
> 
> How it works is that when PPE decides it needs to tunnel the packet,
> you're going to send the packet back to MQ via SERDES (which will then
> send it again to some PPE, not the same). I think what that bandwidth
> command does is change the stream allocation, you should see it in
> 'show <MQ/XM...> <#> stream'.
> 
> In theory, because PPE can process packet forever (well, until
> watchdog kills the PPE for thinking it is stuck) you could very
> cheaply do outer+inner at the local PPE, but I think that would mean
> that certain features like QoS would not work on the inner interface,
> so I think all this expensive recirculation and SERDES consumption is
> to satisfy quite limited need, and it should be possible to implement
> some 'performance mode' for tunneling, where these MQ/XM provided
> features are not available, but performance cost in most cases is
> negligible.
> 
> In parallel to opening the JTAC case, you might want to try to
> experiment in which FPC/PIC you set the tunneling bandwidth to. I
> don't understand how the tunneling would work if the MQ/XM is remote,
> like would you then also steal fabric capacity every time you tunnel,
> not just  MQ>LU>MQ>LU SERDES, but MQ>LU>MQ>FAB>MQ>LU. So intuitively I
> would recommend ensuring you have the bandwidth configured at the
> local PFE, if you don't know what the local PFE is, just configure it
> everywhere?
> Also you could consult several counters to see if some stream or
> fabric is congested, and these tunneled packets are being sent over
> congested fabric every time with lower fabric qos.
> 
> I don't understand why the bandwidth command is a thing, and why you
> can configure where it is. To me it seems obvious they should always
> handle tunneling strictly locally, never over fabric, because you
> always end up stealing more capacity if you send it to remote MQ. That
> is, implicitly it should be on for every MQ, and every PPE tunnel via
> local MQ.
> 
> -- 
>  ++ytti