Nationwide Routing issues with Wiltel

Wed Jun 28 20:39:12 UTC 2006

On Tue, 27 Jun 2006, Jeremy Chadwick wrote:

> Was anyone able to get an RFO or post-mortem for this?

"An inaccurate set of BGP policies were distributed to routers connected 
to AS791 1 when an automated update script ran at 1100 MDT.  The update 
script regularly ran every two hours to update the network with current 
BGP information. Due to the scheduled shutdown of the legacy BGP policy 
server and subsequent con version to the Level3 route registry engine, the 
old server policy server was sh utdown. In addition, the scripts used to 
update routes on the network were to be disabled. One of these scripts 
wasn t disabled as intended.  As a result, the script ran as scheduled at 
1300MDT and consequently pushed partial configurations to production 
routers because the script was unable to communicate with decommissioned 
policy server.  Incorrect policies were exchanged between AS7911 s 
customers and peers resulted in increased latency; as large route blocks 
attem pted to traverse individual customer connections.

Repair

Updated configurations were pushed to all the routers, individual 
connections were cleaned up and BGP sessions were restored.  In addition, 
the automated BGP script has been shut-off.  Maximum pre-fix list limits 
have been established across the network as a risk mitigation step.
"

---
david raistrick        http://www.netmeister.org/news/learn2quote.html
drais at atlasta.net             http://www.expita.com/nomime.html