Verizon connectivity issues

Mon Aug 25 17:54:17 UTC 2014

Hello all,

We have heard from many of our customers in the Northeast region,
specifically PA, MD, and VA who have difficulty connecting to our website
(very slow loading times).  We have noticed that if our data center
provider specifically routes our outbound traffic over Hurricane Electric
rather than XO Communications, that connectivity is restored to normal for
our customers.  The HE path goes from Salt Lake City to Denver where it is
handed off to TeliaSonera, and they send it to Chicago where it gets handed
off to Verizon.  This works great and generally has a ping response of
20-25ms less than the bad route as posted below from one of our servers:

Tracing route to pool-108-52-xxx-xxx.phlapa.fios.verizon.net
 [108.52.xxx.xxx]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  172.xxx.xxx.xxx (Internal IP)
  2    <1 ms    10 ms    <1 ms  209-41-xxx-xxx.c7dc.com [209.41.xxx.xxx]
  3    <1 ms    <1 ms    <1 ms  e1-5.rt08.gp1.c7dc.com [192.41.0.97]
  4     2 ms     1 ms     1 ms  ip65-46-51-113.z51-46-65.customer.algx.net
 [65.46.51.113]
  5    19 ms    18 ms    19 ms  vb1611.rar3.sanjose-ca.us.xo.net
 [216.156.0.5]
  6    21 ms    18 ms    18 ms  207.88.14.226.ptr.us.xo.net [207.88.14.226]
  7    18 ms    18 ms    18 ms  206.111.6.122.ptr.us.xo.net [206.111.6.122]
 >>>> Verizon owned device with an XO IP address
  8    87 ms    86 ms    87 ms  b100.phlapa-lcr-22.verizon-gni.net
[130.81.209.187]
 >>>> Next hop goes from CA to PA, probably just hidden routing info
  9     *        *        *     Request timed out.
 10    88 ms    89 ms    90 ms  pool-108-52-xxx-xxx.phlapa.fios.verizon.net
 [108.52.xxx.xxx]

As you can see, XO sends this traffic from Salt Lake City to San Jose where
it is handed off to Verizon.  We've worked with XO and determined that
there are no connectivity issues along their path, which seems to point to
an issue within Verizon's network.  I have a couple of tickets open with
Verizon at the moment on behalf of some of our customers, but we have had
trouble breaking through Tier 1.

Our first thought was saturation of the peering point, but XO states that
the link between hops 6 and 7 is using 5Gbps out of 20Gbps capacity, and
the latency on hop 7 (the Verizon owned device with an XO IP) seems normal
whenever we test.

Any Verizon engineers out there willing to take a quick peek at this route
for any obvious issues?  It would be a lot easier for us if Verizon
provided Looking Glass servers, but alas...

Thank you,
Ryan