Verizon connectivity issues
Bacon, Ricky (RJ)
rj.bacon at verizon.com
Thu Aug 28 14:18:19 UTC 2014
@Ryan, XO was accurate, the peering connection between 6 and 7 is not congested, nor is it taking any sort of errors.
I took a look. The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of multiple physical routers). That hop traverses the continental US from San Jose to Philadelphia, so you would expect that, and I don't see anything going on in that path. The end to end rtt is pretty normal, and in any case, it shouldn't prevent your customers from connecting. I am not seeing any packet loss on that path. It's possible that whatever they saw was a transient issue and may be over now.
So, please double check with your customers to verify that they still have problems. If so, continue to escalate the ticket.
From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Christopher Morrow
Sent: Wednesday, August 27, 2014 10:52 PM
To: ryanruuska+nanog at gmail.com
Cc: nanog list
Subject: Re: Verizon connectivity issues
On Mon, Aug 25, 2014 at 1:54 PM, Ryan Ruuska <ryanruuska+nanog at gmail.com> wrote:
> Hello all,
> We have heard from many of our customers in the Northeast region,
> specifically PA, MD, and VA who have difficulty connecting to our
there's probably vz customer folk on-list, maybe 'what should they test for you' would be nice? :)
> (very slow loading times). We have noticed that if our data center
> provider specifically routes our outbound traffic over Hurricane
> Electric rather than XO Communications, that connectivity is restored
> to normal for our customers. The HE path goes from Salt Lake City to
> Denver where it is handed off to TeliaSonera, and they send it to
> Chicago where it gets handed off to Verizon. This works great and
> generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers:
> Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net
> over a maximum of 30 hops:
> 1 <1 ms <1 ms <1 ms 172.xxx.xxx.xxx (Internal IP)
> 2 <1 ms 10 ms <1 ms 209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx]
> 3 <1 ms <1 ms <1 ms e1-5.rt08.gp1.c7dc.com [18.104.22.168]
> 4 2 ms 1 ms 1 ms ip65-46-51-113.z51-46-65.customer.algx.net
> 5 19 ms 18 ms 19 ms vb1611.rar3.sanjose-ca.us.xo.net
> 6 21 ms 18 ms 18 ms 22.214.171.124.ptr.us.xo.net [126.96.36.199]
> 7 18 ms 18 ms 18 ms 188.8.131.52.ptr.us.xo.net [184.108.40.206]
> >>>> Verizon owned device with an XO IP address
> 8 87 ms 86 ms 87 ms b100.phlapa-lcr-22.verizon-gni.net
> >>>> Next hop goes from CA to PA, probably just hidden routing info
> 9 * * * Request timed out.
> 10 88 ms 89 ms 90 ms pool-108-52-xxx-xxx.phlapa.fios.verizon.net
> As you can see, XO sends this traffic from Salt Lake City to San Jose
> where it is handed off to Verizon. We've worked with XO and
> determined that there are no connectivity issues along their path,
> which seems to point to an issue within Verizon's network. I have a
> couple of tickets open with Verizon at the moment on behalf of some of
> our customers, but we have had trouble breaking through Tier 1.
> Our first thought was saturation of the peering point, but XO states
> that the link between hops 6 and 7 is using 5Gbps out of 20Gbps
> capacity, and the latency on hop 7 (the Verizon owned device with an
> XO IP) seems normal whenever we test.
> Any Verizon engineers out there willing to take a quick peek at this
> route for any obvious issues? It would be a lot easier for us if
> Verizon provided Looking Glass servers, but alas...
they have route data sent to routeviews at least...
More information about the NANOG