Verizon: Extremely Strange CPE Routing in NYC/NJ Area
Matthew Huff
mhuff at ox.com
Fri Nov 30 13:17:17 UTC 2018
Windows traceroute uses ICMP by default, Unix based systems use UDP. For example from my Mac on Fios in Westchester, NY (and yes, they are doing something with ICMP traceroutes, possibly within their gateway box):
acpro:Documents mhuff$ sudo traceroute www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.9
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.9), 64 hops max, 52 byte packets
1 firewall (10.1.1.1) 0.916 ms 0.419 ms 0.394 ms
2 * * *
3 b3369.nycmny-lcr-22.verizon-gni.net (130.81.16.102) 3.939 ms 10.172 ms
b3369.nycmny-lcr-21.verizon-gni.net (130.81.16.100) 5.192 ms
4 * * *
5 0.ae4.xl2.buf4.alter.net (140.222.228.143) 13.500 ms * 12.970 ms
6 0.xe-5-3-0.gw1.buf4.alter.net (152.63.26.62) 12.390 ms 12.122 ms
0.ae4.xl2.buf4.alter.net (140.222.228.143) 12.704 ms
7 0.xe-4-2-0.gw1.buf4.alter.net (152.63.26.46) 14.326 ms 13.069 ms
verizon.com.customer.alter.net (204.148.47.162) 14.436 ms
8 et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103) 13.990 ms
unknown-72-30-223-x.yahoo.com (72.30.223.3) 14.220 ms
et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103) 13.711 ms
9 et-0-0-0.msr2.bf1.yahoo.com (74.6.227.137) 12.098 ms
et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129) 13.388 ms
et-18-0-1.msr1.bf1.yahoo.com (74.6.227.131) 14.457 ms
10 et-19-0-0.clr2-a-gdc.bf1.yahoo.com (74.6.122.37) 14.554 ms
et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129) 14.810 ms
et-1-1-1.msr1.bf1.yahoo.com (74.6.227.135) 13.502 ms
11 eth-18-3.bas2-1-flk.bf1.yahoo.com (98.139.128.75) 14.607 ms 16.608 ms
eth-18-3-bas1-1-flk.bf1.yahoo.com (98.139.128.73) 15.161 ms
12 media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9) 13.470 ms
eth-17-3.bas2-1-flk.bf1.yahoo.com (98.139.128.71) 13.582 ms
media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9) 12.952 ms
macpro:Documents mhuff$ sudo traceroute -I www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.10
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.10), 64 hops max, 72 byte packets
1 firewall (10.1.1.1) 0.675 ms 0.347 ms 0.322 ms
2 media-router-fp2.prod1.media.vip.bf1.yahoo.com (72.30.35.10) 2.456 ms 21.139 ms 12.834 ms
----
Matthew Huff | 1 Manhattanville Rd
Director of Operations | Purchase, NY 10577
OTA Management LLC | Phone: 914-460-4039
-----Original Message-----
From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Lee
Sent: Thursday, November 29, 2018 5:42 PM
To: Nick Zurku <nzurku at teraswitch.com>
Cc: Brendan Mannella <bmannella at teraswitch.com>; nanog at nanog.org; Todd Kammerer <todd at libsyn.com>
Subject: Re: Verizon: Extremely Strange CPE Routing in NYC/NJ Area
On 11/29/18, Nick Zurku <nzurku at teraswitch.com> wrote:
> Can anyone from Verizon take a look at this behavior for us?
>
> We’re having multiple Verizon FiOS users in the NYC/NJ area appear to
> teleport from their FiOS router to our IP in the Pittsburgh region.
Verizon is doing something seriously weird to windows traceroute:
C:\Users\Lee>tracert www.yahoo.com
Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms fw.home.net
2 1 ms <1 ms <1 ms vbz-router.home.net [192.168.1.1]
3 8 ms 3 ms 6 ms
media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232]
Trace complete.
C:\Users\Lee>ping -i 12 www.yahoo.com.
Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Ping statistics for 98.138.219.232:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
C:\Users\Lee>ping -i 13 www.yahoo.com.
Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.219.232: bytes=32 time=32ms TTL=54 Reply from 98.138.219.232: bytes=32 time=31ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54
Ping statistics for 98.138.219.232:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds:
Minimum = 31ms, Maximum = 33ms, Average = 32ms
C:\Users\Lee>
traceroute from a linux box works as expected..
Lee
> Users
> are seeing extreme slowness with TCP traffic, but ping times seem
> reasonable.
>
> User 1:
> 1 fios_quantum_gateway (192.168.1.1) 1.575 ms 2.426 ms 3.193 ms
> 2 204.16.244.8 (204.16.244.8) 2.269 ms 3.055 ms 2.727 ms
>
> User 2:
> 1 fios_quantum_gateway (192.168.1.1) 1.565 ms 1.048 ms 0.947 ms
> 2 204.16.244.8 (204.16.244.8) 2.162 ms 3.588 ms 3.048 ms
>
> I can provide end-user NYC/NJ IPs off-list if desirable.
>
>
> Here's a normal looking trace from an FiOS line locally in the
> Pittsburgh
> region:
>
>
> IP: 108.39.229.34
> Tracing route to four.libsyn.com [204.16.244.8] over a maximum of 30
> hops:
>
> 1 <1 ms <1 ms <1 ms 192.168.1.1
> 2 5 ms 2 ms 7 ms lo0-100.PITBPA-VFTTP-301.verizon-gni.net
> <http://lo0-100.pitbpa-vfttp-301.verizon-gni.net/> [108.39.229.1]
> 3 5 ms 6 ms 6 ms B3301.PITBPA-LCR-22.verizon-gni.net
> <http://b3301.pitbpa-lcr-22.verizon-gni.net/> [100.41.223.244]
> 4 * * * Request timed out.
> 5 * * * Request timed out.
> 6 13 ms 12 ms 13 ms 0.et-7-1-5.BR1.IAD8.ALTER.NET
> <http://0.et-7-1-5.br1.iad8.alter.net/> [140.222.226.17]
> 7 10 ms 10 ms 10 ms verizon.com.customer.alter.net
> [152.179.50.110]
> 8 12 ms 12 ms 13 ms be3084.ccr42.dca01.atlas.cogentco.com
> [154.54.30.65]
> 9 22 ms 22 ms 22 ms be2820.rcr21.pit02.atlas.cogentco.com
> [154.54.83.54]
> 10 22 ms 22 ms 21 ms 38.104.120.90
> 11 26 ms 21 ms 19 ms 204.16.241.133
> 12 * * * Request timed out.
> 13 21 ms 21 ms 21 ms 204.16.244.8
>
> Is this a possible traffic engineering blip? I can’t say we’ve ever
> seen trace routes return such sparse results and actually make it to
> the destination.
>
> --
> Nick Zurku
> Systems Engineer
> TeraSwitch, Inc.
> Cell: 412-953-0481
> Office: 412-945-7048
> nzurku at teraswitch.com
>
More information about the NANOG
mailing list