<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello all.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Over time, we've run into occurrences of both bugs and human error, both in our own gear and in our partner networks' gear,
<b>specifically affecting multi-path forwarding</b>, at pretty much all layers: Multi-chassis LAG, ECMP, and BGP MP. (Yes, I am a corner-case magnet. Lucky me.)</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Some of these issues were fairly obvious when they happened, but some were <i>really</i> hard to pin down.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We've found that typical network monitoring tools (Observium & Smokeping, not to mention plain old ping and traceroute) can't really detect a hashing-related or multi-path-related problem: either the packets get through or they don't.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Can anyone recommend either tools or techniques to validate that multi-path forwarding either is, or isn't, working correctly in a production network? I'm looking to add something to our test suite for when we make changes to critical network gear. Almost
all the scenarios I want to test only involve two paths, if that helps.<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The best I've come up with so far is to have two test systems (typically VMs) that use adjacent IP addresses and adjacent MAC addresses, and test both inbound and outbound to/from those, blindly trusting/hoping that hashing algorithms will
<i>probably</i> exercise both paths.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Some of the problems we've seen show that merely looking at interface counters is insufficient, so I'm trying to find an explicit proof, not implicit.<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Any suggestions? Surely other vendors and/or admins have screwed this up in subtle ways enough times that this knowledge exists? (My Google-fu is usually pretty good, but I'm striking out - maybe I'm using the wrong terms.)<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-Adam<br>
</div>
<div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<b style="text-align:start; background-color:rgb(255,255,255)"><span style="margin:0px; font-size:10pt; color:rgb(68,84,106)">Adam Thompson</span></b><span style="margin:0px; font-size:9pt; color:rgb(68,84,106); text-align:start; background-color:rgb(255,255,255)"><br>
Consultant, Infrastructure Services<br>
<img alt="1593169877849" style="margin:0px; max-width:100%; height:auto" data-outlook-trace="F:2|T:2" src="cid:e96ca892-9432-496c-b9c0-0e735f222825"><br>
100 - 135 Innovation Drive<br>
Winnipeg, MB, R3T 6A8<br>
(204) 977-6824 or 1-800-430-6404 (MB only)<br>
<a href="mailto:athompson@merlin.mb.ca" target="_blank" rel="noopener noreferrer" style="margin:0px"><span style="margin:0px">athompson@merlin.mb.ca</span></a><br>
<a href="http://www.merlin.mb.ca/" target="_blank" rel="noopener noreferrer" style="margin:0px"><span style="margin:0px">www.merlin.mb.ca</span></a></span><br>
</div>
</div>
</div>
</div>
</body>
</html>