Verizon connectivity issues

Bacon, Ricky (RJ) rj.bacon at
Thu Aug 28 14:18:19 UTC 2014

Hi Chris!

@Ryan, XO was accurate, the peering connection between 6 and 7 is not congested, nor is it taking any sort of errors.  

I took a look.  The significant increase in rtt seems to happen between 7 and 8 (an Verizon LSP consisting of multiple physical routers).  That hop traverses the continental US from San Jose to Philadelphia, so you would expect that, and I don't see anything going on in that path.  The end to end rtt is pretty normal, and in any case, it shouldn't prevent your customers from connecting.  I am not seeing any packet loss on that path.  It's possible that whatever they saw was a transient issue and may be over now.

So, please double check with your customers to verify that they still have problems.  If so, continue to escalate the ticket.  

-- rj

-----Original Message-----
From: NANOG [mailto:nanog-bounces at] On Behalf Of Christopher Morrow
Sent: Wednesday, August 27, 2014 10:52 PM
To: ryanruuska+nanog at
Cc: nanog list
Subject: Re: Verizon connectivity issues

On Mon, Aug 25, 2014 at 1:54 PM, Ryan Ruuska <ryanruuska+nanog at> wrote:
> Hello all,
> We have heard from many of our customers in the Northeast region, 
> specifically PA, MD, and VA who have difficulty connecting to our 
> website

there's probably vz customer folk on-list, maybe 'what should they test for you' would be nice? :)

> (very slow loading times).  We have noticed that if our data center 
> provider specifically routes our outbound traffic over Hurricane 
> Electric rather than XO Communications, that connectivity is restored 
> to normal for our customers.  The HE path goes from Salt Lake City to 
> Denver where it is handed off to TeliaSonera, and they send it to 
> Chicago where it gets handed off to Verizon.  This works great and 
> generally has a ping response of 20-25ms less than the bad route as posted below from one of our servers:
> Tracing route to
>  []
> over a maximum of 30 hops:
>   1    <1 ms    <1 ms    <1 ms (Internal IP)
>   2    <1 ms    10 ms    <1 ms []
>   3    <1 ms    <1 ms    <1 ms []
>   4     2 ms     1 ms     1 ms
>  []
>   5    19 ms    18 ms    19 ms
>  []
>   6    21 ms    18 ms    18 ms []
>   7    18 ms    18 ms    18 ms []
>  >>>> Verizon owned device with an XO IP address
>   8    87 ms    86 ms    87 ms
> []
>  >>>> Next hop goes from CA to PA, probably just hidden routing info
>   9     *        *        *     Request timed out.
>  10    88 ms    89 ms    90 ms
>  []
> As you can see, XO sends this traffic from Salt Lake City to San Jose 
> where it is handed off to Verizon.  We've worked with XO and 
> determined that there are no connectivity issues along their path, 
> which seems to point to an issue within Verizon's network.  I have a 
> couple of tickets open with Verizon at the moment on behalf of some of 
> our customers, but we have had trouble breaking through Tier 1.
> Our first thought was saturation of the peering point, but XO states 
> that the link between hops 6 and 7 is using 5Gbps out of 20Gbps 
> capacity, and the latency on hop 7 (the Verizon owned device with an 
> XO IP) seems normal whenever we test.
> Any Verizon engineers out there willing to take a quick peek at this 
> route for any obvious issues?  It would be a lot easier for us if 
> Verizon provided Looking Glass servers, but alas...

they have route data sent to routeviews at least...

More information about the NANOG mailing list