Famous operational issues

Sat Mar 13 06:07:43 UTC 2021

Hardly famous and not service-affecting in the end, but figured I'd 
share an incident from our side that occurred back in 2018.

While commissioning a new node in our Metro-E network, an IPv6 
point-to-point address was mis-typed. Instead of ending in /126, it 
ended in /12. This happened in Johannesburg.

We actually came across this by chance while examining the IGP table of 
another router located in Slough, and found an entry for 2c00::/12 
floating around. That definitely looked out of place, as we never carry 
parent blocks in our IGP.

Running the trace from Slough led us back to this one Metro-E device in 
Jo'burg.

It took everyone nearly an hour to figure out the typo, because for all 
the laser focus we had on the supposed link of the supposed box that was 
creating this problem, we all overlooked the fact that the /12 
configured on the point-to-point link was actually supposed to have been 
a /126.

The reason this never caused a service problem was because we do not 
redistribute our IGP into BGP (not that anyone should). And even if we 
did, there are a ton of filters and BGP communities on all devices to 
ensure a route such as that would have never made it out of our AS.

Also, the IGP contains the most specific paths to every node in our 
network, so the presence of the 2c00::/12 was mostly cosmetic. It would 
have never been used for routing decisions.

Mark.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210313/1974eca2/attachment.html>