<div dir="ltr">Nail on the head, David.<div><br></div><div>I've got ~3 dozen IPSEC connections that land back at our DC. Only a few were affected. Some physically near each other...etc No rhyme or reason to the selection. So LAG hashing makes perfect sense. Due to the moderator queue, my message was delayed. It started around Noon Saturday. As of this morning (monday), It's resolved. I did spend about an hour trying to get a ticket in. And did, but never heard back. Took me about a half-hour to convince the rep I didn't want someone dispatched.</div><div><br></div><div>Yes, In this case the AS path was 5650>6939>Me. However, The return is Me>174>3356>5650. 5650 and 6939 peer across the FL-IX in MIA. But 6939 isn't accepting any routes from 5650 on that session. After talking to HE about it, They claim it's intentionally filtered due to capacity issues on the port, Waiting for 5650 to augment. Sounds like the classic peering staring contest to me. It was nice for a while. And kept us out of trouble during the last "nationwide" level 3 outage. As all other paths crossed 3356.</div><div><br></div><div>Given the outbound route from the Frontier circuit didn't touch level 3 (in my case). And that's where my loss was occurring (What left the CPE, never made it to my 6939 port). It doesn't look like this was specific to a LAG between 5650 and 3356 though.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jan 24, 2021 at 9:19 PM David Hubbard <<a href="mailto:dhubbard@dino.hostasaurus.com">dhubbard@dino.hostasaurus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="gmail-m_-6150757285285764956WordSection1">
<p class="MsoNormal">Yes, exactly same issue for us, and it has happened in the past a few years ago fortunately. Any chance the route takes a Level 3 (3356) path? I’m just theorizing here, but my belief is they have some kind of link aggregation in the path
from TB to 3356 (or maybe just internal near some edge) and some traffic is getting hashed onto a problematic link/interface/linecard, etc. where IPSec gets dropped. One of our locations lost IPSec ability to some normal VPN endpoints but not others. And
here’s why I think this is the issue…. if you change the source and/or destination IP address by one, you may find some or all of your sessions magically work again.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">In our case, one of our office locations has a static assignment of (fortunately) five IP’s. We only have one external exposed, four site to site VPN’s. Two began failing Saturday morning. I moved the office firewall’s external IP minus
1 and that fixed both, but broke one that had been fine. On the remote end fortunately I have equipment that’s able to override the local IP for VPN traffic, so without impacting other things it talks to, I was able to add a new IP one off from the previous,
and use that for traffic just to this office location; that fixed the remaining issue.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">If I’d not seen this previously several years ago, and wasted who knows how many hours trying to figure it out, it would have once again taken forever to resolve. Trying to get through their support layer to someone who can really help
is impossible. The support is really complete garbage at this point after the Verizon dump; I was going to say service, but that’s been stable outside of these random weird issues that are impossible to resolve with support.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I tried to be a nice guy and raise this through the support channels, but could not make it past the layer where they want me to take our office down to have someone plug a laptop in with our normal WAN IP and “prove” ipsec isn’t working
with different equipment. I was like dude I just told you what I did to get it working again, offered packet captures, just escalate it, but ultimately gave up and hung up.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">David<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">NANOG <nanog-bounces+dhubbard=<a href="mailto:dino.hostasaurus.com@nanog.org" target="_blank">dino.hostasaurus.com@nanog.org</a>> on behalf of Nick Olsen <<a href="mailto:nick@141networks.com" target="_blank">nick@141networks.com</a>><br>
<b>Date: </b>Sunday, January 24, 2021 at 8:42 PM<br>
<b>To: </b>"<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>" <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>><br>
<b>Subject: </b>Frontier Tampa issues<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Anyone else seeing weird things on Tampa/Bradenton FIOS connections?<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've got three unrelated customers that cant establishes IPsec back to me.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">And a third that can't process credit cards out to their third party merchant.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Customers are in <a href="http://47.196.0.0/14" target="_blank">47.196.0.0/14</a>.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In All instances, I see the traffic leave the CPE behind the FIOS circuit. The IPSEC traffic never makes it to my DC. And no clue on the credit card traffic. But it goes un-ack'd<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">And just now a fifth has appeared that can't query DNS against 8.8.8.8. Responses go out and never come back.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The first four all started around noon today.<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div>