<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Yeah, could have been one of those...gone from bad to worse things like Dave mentioned... initial problem and course of action perhaps led to a worse problem.<div><br></div><div>I’ve had DWDM issues that have taken down multiple locations far apart from each other due to how the transport guys hauled stuff </div><div><br></div><div>A few years back I had about 15 routers all reboot suddenly... they were all far apart from each other, turned out to be one of the dual bgp sessions to rr cluster flapped and all 15 routers crash rebooted.</div><div><br></div><div>But ~50 hours of downtime !? <br><br><div id="AppleMailSignature" dir="ltr">Aaron</div><div dir="ltr"><br>On Dec 31, 2018, at 11:41 AM, Dave Temkin <<a href="mailto:dave@temk.in">dave@temk.in</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr">On Mon, Dec 31, 2018 at 11:33 AM Naslund, Steve <<a href="mailto:SNaslund@medline.com">SNaslund@medline.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-947870517607139292WordSection1">
<p class="MsoNormal"><span style="color:rgb(31,73,125)">They shouldn’t need OOB to operate existing lambdas just to configure new ones. One possibility is that the management interface also handles master timing which would be a really bad idea but possible (should
be redundant and it should be able to free run for a reasonable amount of time). The main issue exposed is that obviously the management interface is critical and is not redundant enough. That is if we believe the OOB explanation in the first place (which
by the way is obviously not OOB since it wiped out the in band network when it failed).</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Steven Naslund<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chicago IL<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> </span></p></div></div></blockquote><div> </div><div>A theory, and only a theory, is that they decided to, in order to troubleshoot a much smaller problem (OOB/etc.), deploy an optical configuration change that, when faced with inaccessibility to multiple nodes, ended up causing a significant inconsistency in their optical network, wreaking havoc on all sorts of other systems. With the OOB network already in chaos, card reseats were required to stabilize things on that network and then they could rebuild the optical network from a fully reachable state.</div><div><br></div><div>Again, only a theory.</div><div><br></div><div>-Dave</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_-947870517607139292WordSection1"><p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(31,73,125)">></span>This seems entirely plausible given that DWDM amplifiers and lasers being a complex analog system, they need OOB to align. <u></u><u></u></p>
<div id="gmail-m_-947870517607139292AppleMailSignature">
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">></span>--<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">></span>Eric<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
<br>
<u></u><u></u></p>
</div>
</div>
</div>
</blockquote></div></div>
</div></blockquote></div></body></html>