Strange behavior on the Juniper MX240

Thu May 5 19:50:18 UTC 2022

Friend of mine had this issue recently on an MX chassis running DPC's and RE-2000's.

The extend memory command others have mentioned worked for him.

His instance drove us crazy for a bit. The device would learn a route, show that it was installed (show routes) but traffic to said prefix would bounce net unreachable. We even pushed a static just for S&G's and that still didn't resolve it. It was a single prefix that a customer had reported.

Some things to consider, as others have mentioned.

  1.  IPv6 routes share the same space. And use more per-route. You can extend the life of this box (probably considerably) by dropping full tables for IPv6. Perhaps taking just a default (Same goes for v4).
  2.  It seems from your previous output that you're taking ~1 full v4 table. And 2x v6 tables. Do you really need a full table if you're only taking 1 v4 table? Consider switching to a default only? In my Colleagues case, he was taking 2 full tables of v4 and v6 until he hit the same issue.
  3.  While you're RE's could use a nice upgrade too. Your linecards are actually the problem here. If you move to anything > DPC you get the trio chipset with much more FIB space (2 Million routes I believe?). I'd consider new RE's and new line cards for this box. Which might also mean new switch fabric controllers.... Basically, we'd be talking a full overhaul sans the power supplies and chassis.
  4.  Consider taking a default + full routes. Then filtering > /24 (if you even have anything < /24 learned now) (/48 on IPv6).

Start with the memory command first and see where that gets you. But keep a watchful eye out for this to happen again (as the DFZ grows). Eventually your only option will be to filter routes and rely on a default or upgrade.
________________________________
From: NANOG <nanog-bounces+nick=141networks.com at nanog.org> on behalf of Nehul Patel <nehul.patel at gmail.com>
Sent: Wednesday, May 4, 2022 3:56 PM
To: nanog at nanog.org <nanog at nanog.org>
Subject: Strange behavior on the Juniper MX240

Hi NANOG,

We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis

If Someone has seen these errors before please suggest how to resolve it

May  4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K
May  4 12:42:01   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May  4 12:42:01   /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May  4 12:42:01   last message repeated 4 times
May  4 12:42:01   fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed)
May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576
May  4 12:42:01   fpc0 RT-HAL,rt_msg_handler,540: route process failed
May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0
May  4 12:42:01   fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed)
May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576
May  4 12:42:01   fpc0 RT-HAL,rt_msg_handler,540: route process failed
May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0
May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0
May  4 12:42:01   /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May  4 12:42:01   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May  4 12:42:01   /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid)
May  4 12:42:02   fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0
May  4 12:42:02   fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed)
May  4 12:42:02   fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May  4 12:42:02   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583
May  4 12:42:02   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May  4 12:42:02   fpc0 RT-HAL,rt_msg_handler,540: route process failed

May  4 09:33:17   fpc0 RSMON: Resource Category:jtree  Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:17   fpc0 RSMON: Resource Category:jtree  Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit()
May  4 09:33:18   fpc0 RSMON: Resource Category:jtree  Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:18   fpc0 RSMON: Resource Category:jtree  Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May  4 09:33:18   fpc1 RSMON: Resource Category:jtree  Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:18   fpc1 RSMON: Resource Category:jtree  Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit()
May  4 09:33:18   fpc1 RSMON: Resource Category:jtree  Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:18   fpc1 RSMON: Resource Category:jtree  Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May  4 09:33:18   fpc1 RSMON: Resource Category:jtree  Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:19   fpc1 RSMON: Resource Category:jtree  Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May  4 09:33:19   fpc1 RSMON: Resource Category:jtree  Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:19   fpc1 RSMON: Resource Category:jtree  Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit()
May  4 09:33:19   fpc2 RSMON: Resource Category:jtree  Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:19   fpc2 RSMON: Resource Category:jtree  Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May  4 09:33:19   fpc2 RSMON: Resource Category:jtree  Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May  4 09:33:19   fpc2 RSMON: Resource Category:jtree  Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()

Any suggestions will be helpful

Please do let me know if you have any questions.

Regards and thanks,
 Nehul

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20220505/e8345972/attachment.html>