BGP convergence problem
Andy B.
globichen at gmail.com
Tue Jun 8 09:58:42 UTC 2010
Hi,
This morning there was an ethernet loop problem on DECIX, causing many
BGP sessions to flap throughout the entire platform.
While this can happen, I am myself facing with BGP convergence
problems on our DECIX router (SUP720-3BXL with IOS SXI3).
De DECIX loop has been solved two hours ago, but my BGP sessions are
still flapping and not converging at all. This has been flooding our
logs, and is still going on:
Jun 8 11:47:03 x.x.x.131 239447: Jun 8 11:48:38.364 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.32 Up
Jun 8 11:47:03 x.x.x.131 239448: Jun 8 11:48:38.364 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.231 Up
Jun 8 11:47:03 x.x.x.131 239449: Jun 8 11:48:38.364 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.109 Up
Jun 8 11:47:03 x.x.x.131 239450: Jun 8 11:48:38.364 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.50 Up
Jun 8 11:47:03 x.x.x.131 239451: Jun 8 11:48:38.364 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.81 Up
Jun 8 11:47:03 x.x.x.131 239452: Jun 8 11:48:38.364 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.28 Up
Jun 8 11:47:03 x.x.x.131 239453: Jun 8 11:48:38.364 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.212 Up
Jun 8 11:47:03 x.x.x.131 239454: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.147 Up
Jun 8 11:47:03 x.x.x.131 239455: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.74 Up
Jun 8 11:47:03 x.x.x.131 239456: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.241 Up
Jun 8 11:47:03 x.x.x.131 239457: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.5 Up
Jun 8 11:47:03 x.x.x.131 239458: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.40 Up
Jun 8 11:47:03 x.x.x.131 239459: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::1A44:0:1 Up
Jun 8 11:47:03 x.x.x.131 239460: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::8605:0:1 Up
Jun 8 11:47:03 x.x.x.131 239461: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::1A0B:0:1 Up
Jun 8 11:47:03 x.x.x.131 239462: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::3029:0:1 Up
Jun 8 11:47:03 x.x.x.131 239463: Jun 8 11:48:38.368 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::6E4:0:1 Up
Jun 8 11:47:03 x.x.x.131 239464: Jun 8 11:48:38.372 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::CB0:0:1 Up
Jun 8 11:47:03 x.x.x.131 239465: Jun 8 11:48:38.372 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::21C8:0:1 Up
Jun 8 11:47:03 x.x.x.131 239466: Jun 8 11:48:38.372 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::8463:0:2 Up
Jun 8 11:47:04 x.x.x.131 239467: Jun 8 11:48:38.372 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::31AA:0:1 Up
Jun 8 11:47:04 x.x.x.131 239468: Jun 8 11:48:38.372 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.29 Up
Jun 8 11:47:04 x.x.x.131 239469: Jun 8 11:48:38.372 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::62BF:0:1 Up
Jun 8 11:47:04 x.x.x.131 239470: Jun 8 11:48:39.656 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.101 Down BGP Notification sent
Jun 8 11:47:04 x.x.x.131 239471: Jun 8 11:48:39.656 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.101 4/0 (hold time
expired) 0 bytes
Jun 8 11:47:07 x.x.x.131 239472: Jun 8 11:48:41.696 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.104 Up
Jun 8 11:47:10 x.x.x.131 239473: Jun 8 11:48:44.488 CEST:
%BGP-3-BGP_NO_REMOTE_READ: 80.81.193.187 connection timed out - has
not accepted a message from us for 20000ms (hold time), 1 messages
pending transmition.
Jun 8 11:47:10 x.x.x.131 239474: Jun 8 11:48:44.488 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.187 Down BGP Notification sent
Jun 8 11:47:10 x.x.x.131 239475: Jun 8 11:48:44.488 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.193.187 4/0 (hold time
expired) 0 bytes
Jun 8 11:47:10 x.x.x.131 239476: Jun 8 11:48:44.900 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.61 Up
Jun 8 11:47:10 x.x.x.131 239477: Jun 8 11:48:44.900 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.149 Up
Jun 8 11:47:10 x.x.x.131 239478: Jun 8 11:48:44.900 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.136 Up
Jun 8 11:47:10 x.x.x.131 239479: Jun 8 11:48:44.904 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::8463:0:1 Up
Jun 8 11:47:10 x.x.x.131 239480: Jun 8 11:48:46.352 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::6268:0:1 Up
Jun 8 11:47:14 x.x.x.131 239481: Jun 8 11:48:48.084 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.78 Up
Jun 8 11:47:14 x.x.x.131 239482: Jun 8 11:48:49.172 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.239 Up
Jun 8 11:47:14 x.x.x.131 239483: Jun 8 11:48:49.172 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.24 Up
Jun 8 11:47:17 x.x.x.131 239484: Jun 8 11:48:52.160 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.45 Up
Jun 8 11:47:17 x.x.x.131 239485: Jun 8 11:48:52.160 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.108 Up
Jun 8 11:47:17 x.x.x.131 239486: Jun 8 11:48:52.160 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.164 Up
Jun 8 11:47:17 x.x.x.131 239487: Jun 8 11:48:52.164 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.49 Up
Jun 8 11:47:17 x.x.x.131 239488: Jun 8 11:48:52.164 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.139 Up
Jun 8 11:47:17 x.x.x.131 239489: Jun 8 11:48:52.164 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::1536:0:1 Up
Jun 8 11:47:17 x.x.x.131 239490: Jun 8 11:48:52.164 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::8601:0:1 Up
Jun 8 11:47:17 x.x.x.131 239491: Jun 8 11:48:53.788 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.45 Up
Jun 8 11:47:17 x.x.x.131 239492: Jun 8 11:48:53.788 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::A2DC:0:1 Up
Jun 8 11:47:21 x.x.x.131 239493: Jun 8 11:48:55.056 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.91 Down BGP Notification sent
Jun 8 11:49:04 x.x.x.131 239583: Jun 8 11:50:37.684 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.14 Down Peer closed the session
Jun 8 11:49:04 x.x.x.131 239584: Jun 8 11:50:38.656 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.120 Down BGP Notification sent
Jun 8 11:49:04 x.x.x.131 239585: Jun 8 11:50:38.656 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.120 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:04 x.x.x.131 239586: Jun 8 11:50:38.656 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.229 Down BGP Notification sent
Jun 8 11:49:04 x.x.x.131 239587: Jun 8 11:50:38.656 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.229 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:04 x.x.x.131 239588: Jun 8 11:50:38.656 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.108 Down BGP Notification sent
Jun 8 11:49:04 x.x.x.131 239589: Jun 8 11:50:38.656 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.108 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:07 x.x.x.131 239590: Jun 8 11:50:41.944 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.73 Down BGP Notification sent
Jun 8 11:49:07 x.x.x.131 239591: Jun 8 11:50:41.944 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.194.73 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:07 x.x.x.131 239592: Jun 8 11:50:41.944 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::20AD:0:2 Down BGP Notification
sent
Jun 8 11:49:07 x.x.x.131 239593: Jun 8 11:50:41.944 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 2001:7F8::20AD:0:2 4/0 (hold
time expired) 0 bytes
Jun 8 11:49:07 x.x.x.131 239594: Jun 8 11:50:41.944 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.115 Down BGP Notification sent
Jun 8 11:49:07 x.x.x.131 239595: Jun 8 11:50:41.944 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.193.115 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:07 x.x.x.131 239596: Jun 8 11:50:44.124 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.3 Down BGP Notification sent
Jun 8 11:49:11 x.x.x.131 239597: Jun 8 11:50:44.124 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.194.3 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:11 x.x.x.131 239598: Jun 8 11:50:45.200 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.215 Down BGP Notification sent
Jun 8 11:49:11 x.x.x.131 239599: Jun 8 11:50:45.200 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.215 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:11 x.x.x.131 239600: Jun 8 11:50:47.336 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.141 Down BGP Notification sent
Jun 8 11:49:11 x.x.x.131 239601: Jun 8 11:50:47.336 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.141 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:14 x.x.x.131 239602: Jun 8 11:50:48.432 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::3B41:0:1 Down BGP Notification
sent
Jun 8 11:49:14 x.x.x.131 239603: Jun 8 11:50:48.432 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 2001:7F8::3B41:0:1 4/0 (hold
time expired) 0 bytes
Jun 8 11:49:14 x.x.x.131 239604: Jun 8 11:50:49.720 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.239 Down BGP Notification sent
Jun 8 11:49:14 x.x.x.131 239605: Jun 8 11:50:49.720 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.239 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:17 x.x.x.131 239606: Jun 8 11:50:50.976 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:2000:3080:B4::1 Down Peer closed the
session
Jun 8 11:49:17 x.x.x.131 239607: Jun 8 11:50:52.976 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.20 Down BGP Notification sent
Jun 8 11:49:17 x.x.x.131 239608: Jun 8 11:50:52.976 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.194.20 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:17 x.x.x.131 239609: Jun 8 11:50:54.044 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.193.21 Down BGP Notification sent
Jun 8 11:49:17 x.x.x.131 239610: Jun 8 11:50:54.044 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.193.21 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:20 x.x.x.131 239611: Jun 8 11:50:56.204 CEST:
%BGP-5-ADJCHANGE: neighbor 2001:7F8::1A0B:0:1 Down BGP Notification
sent
Jun 8 11:49:20 x.x.x.131 239612: Jun 8 11:50:56.204 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 2001:7F8::1A0B:0:1 4/0 (hold
time expired) 0 bytes
Jun 8 11:49:23 x.x.x.131 239613: Jun 8 11:50:58.400 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.63 Down BGP Notification sent
Jun 8 11:49:23 x.x.x.131 239614: Jun 8 11:50:58.400 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.194.63 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:23 x.x.x.131 239615: Jun 8 11:50:59.448 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.97 Down BGP Notification sent
Jun 8 11:49:23 x.x.x.131 239616: Jun 8 11:50:59.448 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.97 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:27 x.x.x.131 239617: Jun 8 11:51:01.664 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.131 Down BGP Notification sent
Jun 8 11:49:27 x.x.x.131 239618: Jun 8 11:51:01.664 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.131 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:27 x.x.x.131 239619: Jun 8 11:51:03.872 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.83 Down BGP Notification sent
Jun 8 11:49:27 x.x.x.131 239620: Jun 8 11:51:03.872 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.194.83 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:27 x.x.x.131 239621: Jun 8 11:51:03.872 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.192.156 Down BGP Notification sent
Jun 8 11:49:30 x.x.x.131 239622: Jun 8 11:51:03.872 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.192.156 4/0 (hold time
expired) 0 bytes
Jun 8 11:49:30 x.x.x.131 239623: Jun 8 11:51:06.056 CEST:
%BGP-5-ADJCHANGE: neighbor 80.81.194.50 Down BGP Notification sent
Jun 8 11:49:30 x.x.x.131 239624: Jun 8 11:51:06.056 CEST:
%BGP-3-NOTIFICATION: sent to neighbor 80.81.194.50 4/0 (hold time
expired) 0 bytes
CPU load is constantly at 100% doing BGP and more BGP.
We have around 200 BGP sessions on DECIX and I would not want to shut
them down and bring them up individually.
How can I get out of this deadlock?
Andy
More information about the NANOG
mailing list