Verizon: Extremely Strange CPE Routing in NYC/NJ Area

Thu Nov 29 22:41:42 UTC 2018

On 11/29/18, Nick Zurku <nzurku at teraswitch.com> wrote:
> Can anyone from Verizon take a look at this behavior for us?
>
> We’re having multiple Verizon FiOS users in the NYC/NJ area appear to
> teleport from their FiOS router to our IP in the Pittsburgh region.

Verizon is doing something seriously weird to windows traceroute:
C:\Users\Lee>tracert www.yahoo.com

Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  fw.home.net
  2     1 ms    <1 ms    <1 ms  vbz-router.home.net [192.168.1.1]
  3     8 ms     3 ms     6 ms
media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232]

Trace complete.

C:\Users\Lee>ping -i 12 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.

Ping statistics for 98.138.219.232:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

C:\Users\Lee>ping -i 13 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.219.232: bytes=32 time=32ms TTL=54
Reply from 98.138.219.232: bytes=32 time=31ms TTL=54
Reply from 98.138.219.232: bytes=32 time=33ms TTL=54
Reply from 98.138.219.232: bytes=32 time=33ms TTL=54

Ping statistics for 98.138.219.232:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 31ms, Maximum = 33ms, Average = 32ms

C:\Users\Lee>

traceroute from a linux box works as expected..

Lee

> Users
> are seeing extreme slowness with TCP traffic, but ping times seem
> reasonable.
>
> User 1:
>  1  fios_quantum_gateway (192.168.1.1)  1.575 ms  2.426 ms  3.193 ms
>  2  204.16.244.8 (204.16.244.8)  2.269 ms  3.055 ms  2.727 ms
>
> User 2:
>  1  fios_quantum_gateway (192.168.1.1)  1.565 ms  1.048 ms  0.947 ms
>  2  204.16.244.8 (204.16.244.8)  2.162 ms  3.588 ms  3.048 ms
>
> I can provide end-user NYC/NJ IPs off-list if desirable.
>
>
> Here's a normal looking trace from an FiOS line locally in the Pittsburgh
> region:
>
>
> IP:  108.39.229.34
> Tracing route to four.libsyn.com [204.16.244.8]
> over a maximum of 30 hops:
>
>  1    <1 ms    <1 ms    <1 ms  192.168.1.1
>  2     5 ms     2 ms     7 ms  lo0-100.PITBPA-VFTTP-301.verizon-gni.net
> <http://lo0-100.pitbpa-vfttp-301.verizon-gni.net/> [108.39.229.1]
>  3     5 ms     6 ms     6 ms  B3301.PITBPA-LCR-22.verizon-gni.net
> <http://b3301.pitbpa-lcr-22.verizon-gni.net/> [100.41.223.244]
>  4     *        *        *     Request timed out.
>  5     *        *        *     Request timed out.
>  6    13 ms    12 ms    13 ms  0.et-7-1-5.BR1.IAD8.ALTER.NET
> <http://0.et-7-1-5.br1.iad8.alter.net/> [140.222.226.17]
>  7    10 ms    10 ms    10 ms  verizon.com.customer.alter.net
>  [152.179.50.110]
>  8    12 ms    12 ms    13 ms  be3084.ccr42.dca01.atlas.cogentco.com
>  [154.54.30.65]
>  9    22 ms    22 ms    22 ms  be2820.rcr21.pit02.atlas.cogentco.com
>  [154.54.83.54]
> 10    22 ms    22 ms    21 ms  38.104.120.90
> 11    26 ms    21 ms    19 ms  204.16.241.133
> 12     *        *        *     Request timed out.
> 13    21 ms    21 ms    21 ms  204.16.244.8
>
> Is this a possible traffic engineering blip? I can’t say we’ve ever
> seen trace routes return such sparse results and actually make it to the
> destination.
>
> --
> Nick Zurku
> Systems Engineer
> TeraSwitch, Inc.
> Cell: 412-953-0481
> Office: 412-945-7048
> nzurku at teraswitch.com
>