Long hops on international paths

Tue Jan 25 16:54:59 UTC 2022

Peering connection, I think, can explain this.
With some notable exceptions (all of whom participate here, I think), carriers still don’t throw around 100G+ peering routers around like sprinkles on a donut.  (Even those big networks don’t, it just looks like they do because they’re freaking huge.)
I suspect what you may be seeing is NOT “international carriers all concentrate on a single router” at all, but rather a) “large carriers tend to interconnect at only a few points” and b) “the best path from North America to Europe will therefore tend to always go through the same small set of inter-AS peering routers on this side”.
What you’ve described, purely in the public-visible layer 3 internet, is normal in my experience.  I would fully expect upwards of 90% of my daily traffic crosses one of 3 peering routers in my network, no surprise there, but ALSO I estimate at least 80% of it crosses one of no more than 5 or 6 routers even as I get 5, 6, 7, 8 hops away from me.  The internet isn’t as diversely-pathed as it once was.
In the MPLS carrier case, I’m aware of a couple that use MPLS to tunnel peering traffic from their edge back to a centralized “core” router that speaks BGP.  Not sure how common this is, I have a very small sample set.  But in this example, no matter how diverse the carrier is at L1/L2, your L3 investigation will always hit the same much-smaller set of routers.

To directly answer your question, the cost/benefit is driven by the fact that running BGP is (relatively) complicated, error-prone and expensive, compared to not running BGP.  And those routers running BGP are (broadly speaking) the routers controlling inter-carrier traffic.  So the “chokepoints” naturally occur as an emergent property of each carrier controlling their own operational and financial risk.  Very much depends on the carrier and their operational philosophy.  I know nearly nothing about Telia, but Zayo doesn’t (didn’t? did they get bought?) have a lot of peering routers – they were historically more of an L2 and/or Private-network operator, as I understand it.  (I was never a customer, so that’s hearsay.)
-Adam

Adam Thompson
Consultant, Infrastructure Services
[MERLIN]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson at merlin.mb.ca<mailto:athompson at merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>

From: NANOG <nanog-bounces+athompson=merlin.mb.ca at nanog.org> On Behalf Of PAUL R BARFORD
Sent: Tuesday, January 18, 2022 8:49 AM
To: davidbass570 at gmail.com
Cc: nanog at nanog.org
Subject: Re: Long hops on international paths

Hello David,

Understanding the physical topology of the network is not our objective.  What we're trying to understand is the logical topology revealed by traceroute (we are well-aware of traceroute limitations) and why a relatively small set of routers in different countries tend to have the majority of the international connections.  Our expectation was that layer 3 connectivity revealed in traceroute to be relatively evenly spread out along coasts and near submarine landing points.  We're not seeing that.  So, the question is what is the cost/benefit to providers to configure/maintain routes (that include long MPLS tunnels) that tend to concentrate international connectivity at a relatively small number of routers?

Regards, PB
________________________________
From: davidbass570 at gmail.com<mailto:davidbass570 at gmail.com> <davidbass570 at gmail.com<mailto:davidbass570 at gmail.com>>
Sent: Tuesday, January 18, 2022 8:22 AM
To: PAUL R BARFORD <pb at cs.wisc.edu<mailto:pb at cs.wisc.edu>>
Cc: morrowc.lists at gmail.com<mailto:morrowc.lists at gmail.com> <morrowc.lists at gmail.com<mailto:morrowc.lists at gmail.com>>; nanog at nanog.org<mailto:nanog at nanog.org> <nanog at nanog.org<mailto:nanog at nanog.org>>
Subject: Re: Long hops on international paths

I think a large part of your problem is that you’re using trace route to try and determine the full topology of a large complex network.  It won’t show the full topology.

On Mon, Jan 17, 2022 at 7:43 PM PAUL R BARFORD <pb at cs.wisc.edu<mailto:pb at cs.wisc.edu>> wrote:
What we're considering specifically are consecutive (layer 3) hops as identified by traceroute.  Thus, TTL is decremented by 1 and no more than 1 (i.e., we have to get full information (not *****) from consecutive hops to consider the link).  I have asked my colleague to put together a set of examples.  We assume that there are multiple layer 1 and 2 links, and possibly layer 3 hops masked from traceroute by MPLS.  But what we're seeing in terms of hops exposed by traceroute make it look like a single (TTL decremented by 1) hop.

I'll post the examples when I get them.

PB
________________________________
From: morrowc.lists at gmail.com<mailto:morrowc.lists at gmail.com> <morrowc.lists at gmail.com<mailto:morrowc.lists at gmail.com>>
Sent: Monday, January 17, 2022 5:13 PM
To: PAUL R BARFORD <pb at cs.wisc.edu<mailto:pb at cs.wisc.edu>>
Cc: Pengxiong Zhu <pzhu011 at ucr.edu<mailto:pzhu011 at ucr.edu>>; nanog at nanog.org<mailto:nanog at nanog.org> <nanog at nanog.org<mailto:nanog at nanog.org>>

Subject: Re: Long hops on international paths

On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD <pb at cs.wisc.edu<mailto:pb at cs.wisc.edu>> wrote:
Dear Pengxiong,

Thanks for your questions:

  1.  We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR alias resolution method to infer IP addresses assigned to the same router.
  2.  We understand the concerns about IP geolocation.  Interfaces of the router in question are assigned similar domain names e.g., “chi-b2-link.ip.twelve99.net<http://chi-b2-link.ip.twelve99.net>” (62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation information, and indicates that this router is located in Chicago.  We cross-reference with Maxmind where possible.  In this particular case, there is the telltale in the use of "chi" in the domain name.
  3.

I think nick's point about ttl expiry and missing some context on topology still stands.
I'd be that the paths between 2 continents do not actually land in chicago... that you're seeing (or not seeing) missing hops between the coast(s) and chicago inside 1299's network in the US.

  1.
Hope that helps.

Regards, PB
________________________________
From: Pengxiong Zhu <pzhu011 at ucr.edu<mailto:pzhu011 at ucr.edu>>
Sent: Monday, January 17, 2022 3:23 PM
To: PAUL R BARFORD <pb at cs.wisc.edu<mailto:pb at cs.wisc.edu>>
Cc: nanog at nanog.org<mailto:nanog at nanog.org> <nanog at nanog.org<mailto:nanog at nanog.org>>
Subject: Re: Long hops on international paths

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based on IP address or MAC addresses? Or using CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in Chicago? IP-geolocation based on rDNS is not always accurate though.

Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD <pb at cs.wisc.edu<mailto:pb at cs.wisc.edu>> wrote:
Hello,

I am a researcher at the University of Wisconsin.  My colleagues at Northwestern University and I are studying international Internet connectivity and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've observed that many international links (i.e., a single hop on an end-to-end path that connects two countries where end points on the hop are identified via rDNS) tend to originate/terminate at the same routers.  Said another way, we are observing a relatively small set of routers in different countries tend to have a majority of the international connections - this is especially the case for hops that terminate in the US.  For example, there is a router operated by Telia (AS1299) in Chicago that has a high concentration of such links.  We were a bit surprised by this finding since even though it makes sense that the set of providers is relatively small (i.e., those that offer global connectivity), we assumed that the set of routers that used for international connectivity within any one country would tend to be more widely distributed (at least with respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, the cost/benefit for configuring international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

--

Regards,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20220125/f40e2502/attachment.html>