Long hops on international paths

Tue Jan 18 14:28:14 UTC 2022

Hey Paul,

> Thank you for the summary.  We're clear about the fact that what we're seeing are MLPS paths - that was not in question.  What we are not clear about and the reason for the post is why the provider - zayo.telia in this case - would decide to configure MPLS paths between Chicago and distant international locations.  We assumed we would see hops in traceroute between Chicago and coastal locations and then hops that transited submarine infrastructure followed by hops to large population centers.

MPLS is the default, not exception.

And like any other form of tunnelling, MPLS decouples underlay and overlay.

This means, the key value proposal of tunneling is that the devices
between tunnel end points do not know the original sender or final
receiver. This means, when TTL expires in transit, the P device may
not know how to return packet to sender.

There are three cases here

1) MPLS-TTL does not expire in transit => easy
2) MPLS-TTL expires in transit
  2a) generate TTL exceeded and put it back to tunnel, sending it to
egressPE, which is guaranteed to know how to return to sender
  2b) randomly assume that you know how to reach the sender and try to
send the TTL exceeded directly

with 2a) all P hops display egressPE latency, but it works. With 2b)
it might not work, as P might not know how to return. Some devices,
like Cisco, allows you heuristically to decide if to tunnel ICMP or
not, based on stack depth, but this does not work. As default table
during repair is as deep as vrf without repair, so we cannot really
discriminate.

So the best solution is to not expire in transit (the norm in
tunneling), i.e. set MPLS-TTL to 255. 2nd best is to tunnel, but the
RTT will confuse uneducated, or as it may be, hjghly educated, users.
We could implement something like
https://ytti.github.io/icmp-eo-timestamp/draft-ytti-intarea-icmp-eo-timestamp.html
to offer correct forward latencies to P/LSR in 2a scenario.

>
> Regards, PB
> ________________________________
> From: Saku Ytti <saku at ytti.fi>
> Sent: Tuesday, January 18, 2022 12:50 AM
> To: PAUL R BARFORD <pb at cs.wisc.edu>
> Cc: Lukas Tribus <lukas at ltri.eu>; Esteban Carisimo <esteban.carisimo at northwestern.edu>; nanog at nanog.org <nanog at nanog.org>; Fabian E. Bustamante <fabianb at cs.northwestern.edu>
> Subject: Re: Long hops on international paths
>
> 1) all (meaning all hitting the zayo.telia) your traceroutes originate
> from University in Chicago
> 2) the zayo.telia device is physically close to the university
> 3) we should expect physically close-by backbone device to be present
> in disproportionate amount of traceroutes
> 4) almost certainly zayo.telia is imposing the MPLS label of TTL 255,
> _NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not
> expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you
> are not seeing any P routers.
>
> This is not esoteric knowledge, but a fairly basic Internet concept. I
> am worried you are missing too much context to produce actionable
> output from your work. It might be interesting to see your curriculum,
> why this confusion arose, why it seems logical that the reason must be
> that almost all waves are terminated there, because it would not seem
> logical for people practising in the field who have even cursory
> understanding, this implies problems in the curriculum.
>
> On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD <pb at cs.wisc.edu> wrote:
> >
> > Please find the examples for the case of Telia below.
> >
> > FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
> >
> >
> >
> > traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
> >
> > 1  216.66.30.101  0.365 ms
> >
> > 2  62.115.49.173  3.182 ms
> >
> > 3  *
> >
> > 4  62.115.137.59  17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> Chicago, IL, US)
> >
> > 5  62.115.117.48  59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> Seattle, WA, US)
> >
> > 6  62.115.171.221  69.993 ms
> >
> > 7  223.120.6.53  69.378 ms
> >
> > 8  223.120.12.34  226.225 ms
> >
> > 9  221.183.55.110  237.475 ms
> >
> > 10  221.183.25.201  238.697 ms
> >
> > 11  221.176.16.213  242.296 ms
> >
> > 12  221.183.36.62  352.695 ms
> >
> > 13  221.183.39.2  300.166 ms
> >
> > 14  117.191.8.118  316.270 ms
> >
> > 15  *
> >
> > 16  *
> >
> > 17  *
> >
> > 18  *
> >
> > 19  *
> >
> >
> >
> >
> >
> > FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
> >
> >
> >
> > traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La Crau, FR)
> >
> > 1  140.192.218.129  0.795 ms
> >
> > 2  140.192.9.124  0.603 ms
> >
> > 3  64.124.44.158  1.099 ms
> >
> > 4  64.125.31.172  3.047 ms
> >
> > 5  *
> >
> > 6  64.125.15.65  1.895 ms      [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
> >
> > 7  62.115.118.59  99.242 ms    [x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC -> Paris, FR)
> >
> > 8  62.115.154.23  105.214 ms
> >
> > 9  77.136.10.6  119.021 ms
> >
> > 10  77.136.10.6  118.830 ms
> >
> > 11  80.118.89.202  118.690 ms
> >
> > 12  80.118.89.234  118.986 ms
> >
> > 13  109.24.108.66  119.159 ms
> >
> > 14  109.25.215.237  126.085 ms
> >
> >
> >
> >
> >
> > traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI)
> >
> > 1  140.192.218.129  0.243 ms
> >
> > 2  140.192.9.124  0.326 ms
> >
> > 3  64.124.44.158  0.600 ms
> >
> > 4  *
> >
> > 5  *
> >
> > 6  64.125.15.65  1.792 ms      [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
> >
> > 7  62.115.123.27  121.199 ms   [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC -> Helsinki, FI)
> >
> > 8  *
> >
> > 9  141.208.193.190  127.723 ms
> >
> > 10  84.249.89.93  139.051 ms
> >
> >
> >
> >
> >
> > traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 193.28.231.50 (MAXMIXD: None, HU)
> >
> > 1  140.192.218.129  0.240 ms
> >
> > 2  140.192.9.124  0.333 ms
> >
> > 3  64.124.44.158  0.648 ms
> >
> > 4  *
> >
> > 5  64.125.25.75  0.752 ms
> >
> > 6  64.125.15.65  1.877 ms      [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
> >
> > 7  62.115.119.39  123.952 ms   [x] (bpt-b2-link.ip.twelve99.net., **I suspect it is in Budapest, HU**)
> >
> > 8  62.115.39.122  117.171 ms
> >
> > 9  88.151.96.148  117.202 ms
> >
> > 10  88.151.96.213  124.787 ms
> >
> > 11  *
> >
> > 12  *
> >
> > 13  *
> >
> > 14  *
> >
> > 15  *
> >
> >
> >
> >
> >
> > traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)
> >
> > 1  140.192.218.129  0.224 ms
> >
> > 2  140.192.9.124  0.545 ms
> >
> > 3  64.124.44.158  0.640 ms
> >
> > 4  *
> >
> > 5  *
> >
> > 6  64.125.15.65  1.786 ms      [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
> >
> > 7  62.115.118.247  54.597 ms   [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC -> Los Angeles, CA, US)
> >
> > 8  62.115.11.129  55.979 ms
> >
> > 9  *
> >
> > 10  *
> >
> > 11  *
> >
> > 12  *
> >
> > 13  *
> >
> >
> >
> >
> >
> > traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)
> >
> > 1  140.192.218.129  2.277 ms
> >
> > 2  140.192.9.124  0.449 ms
> >
> > 3  64.124.44.158  0.576 ms
> >
> > 4  *
> >
> > 5  *
> >
> > 6  64.125.15.65  1.814 ms      [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., CAIDA-GEOLOC -> Chicago, IL, US)
> >
> > 7  62.115.114.41  210.056 ms   [x] (snge-b5-link.ip.twelve99.net.,)
> >
> > 8  62.115.177.11  200.840 ms
> >
> >  9  103.198.140.16  233.636 ms
> >
> > 10  103.198.140.16  232.871 ms
> >
> > 11  103.198.140.171  232.648 ms
> >
> > 12  *
> >
> > 13  *
> >
> > 14  *
> >
> > 15  *
> >
> > 16  *
> >
> >
> > ________________________________
> > From: Lukas Tribus <lukas at ltri.eu>
> > Sent: Monday, January 17, 2022 1:52 PM
> > To: PAUL R BARFORD <pb at cs.wisc.edu>
> > Cc: Nick Hilliard <nick at foobar.org>; nanog at nanog.org <nanog at nanog.org>; Esteban Carisimo <esteban.carisimo at northwestern.edu>; Fabian E. Bustamante <fabianb at cs.northwestern.edu>
> > Subject: Re: Long hops on international paths
> >
> > On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD <pb at cs.wisc.edu> wrote:
> > > What we're curious about is why we're seeing a concentration of hops at a small number of routers that appear on international paths.
> >
> > I suggest you share a few actual examples (IP addresses, traceroutes).
> >
> > I don't think discussing your conclusion based on data we don't have
> > makes sense.
> >
> >
> > Lukas
>
>
>
> --
>   ++ytti

-- 
  ++ytti