Verizon no BGP route to some of AS38365 (182.61.200.0/24)

• Previous message (by thread): Verizon no BGP route to some of AS38365 (182.61.200.0/24)
• Next message (by thread): Verizon no BGP route to some of AS38365 (182.61.200.0/24)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

holow29,

Usually decisions made about limiting route propagation outbound for
certain prefixes tend to happen along financial and operational
constraints.

Remember, outbound route announcements control inbound traffic
volumes.  So, if you've got some full pipes across the ocean, for example,
one way to limit the amount of traffic trying to go across
them is to stop announcing some high-traffic prefixes across them.

Note that it is almost always the route *announcer* that is subtracting
out routes to control traffic volumes; very seldom does the receiver
completely subtract out prefixes heard.  The more usual case is for
the receiver to de-preference the routes down, but to keep them in
their tables as a fallback, because they can control that behaviour.

>From the route-exporting network's perspective, however, they have
no reason to believe that the receiving network would listen to and
obey MEDs that they send, to shift traffic away from the congested
path.  They might have tried adding some prepends along the path,
only to discover the receiving network was using LOCALPREF knob
adjustments to control traffic, which kinda shits all over attempts
to traffic engineer via prepending, at which point the only knob you
have left to turn in order to move traffic off the pathway is to stop
announcing prefixes entirely.

As a not-so-parenthetical aside, this is why I *strongly* encourage
networks that use LOCALPREF as a control knob to limit the depth
to which LOCALPREF adjustments are made; ie, allow overriding
of the default LOCALPREF only to paths that are $AVG_PATH_LENGTH+2
or shorter.  That way, if a peer wants to signal a pathway is congested
and should not be used under normal circumstances, they can still do
that by triple-prepending their ASN.  That way, peers can continue to
announce prefixes, but still have some control over traffic flows via
as-path prepending, versus being backed into the corner of having to
completely stop announcing prefixes entirely across certain peering
sessions.

It is very likely that in this case, CT is not announcing prefixes to AS701
in order to control inbound traffic volumes across certain connections,
because CT has determined that is the only control knob they have
available that works; they have probably already experimented and
found that AS701 squashes inbound MEDs, and uses LOCALPREF
to force traffic flows regardless of as-path prepending.

Your only recourse in a situation like this is likely to be getting
a second connection that bypasses the CT-VZ choke point.

Best of luck!

Matt



On Thu, Jul 21, 2022 at 8:56 AM holow29 <holow29 at gmail.com> wrote:

> In this instance, I would define "correct" behavior as VZ having any route
> to this subnet; after all, customers don't pay VZ to access just their
> network or a subset of the internet. You make a good point, though I would
> expect that if it isn't VZ's business decision to not have this route, they
> would have some sort of vested interest in figuring out why CT is not
> advertising it to them.
> From VZ engineer, I heard that no one is advertising it to AS701, so I
> assume it is not a case of VZ refusing to accept it (which is what I had
> initially assumed having been frustrated in the past with VZ on other
> matters + seeing all their BGP issues in the past few years).
>
> On Thu, Jul 21, 2022 at 11:40 AM Tom Beecher <beecher at beecher.cc> wrote:
>
>> I would expect Verizon to be able to contact CT and figure out why they
>>> aren't passing the *correct* routes just as I might expect Baidu to do
>>> the same thing, I suppose.
>>>
>>
>> What defines 'correct'? ASN's routinely make traffic engineering or
>> business decisions not to announce certain prefixes to certain other ASNs.
>> This certainly does sometimes cause reachability issues for end users, but
>> that's a choice someone made. That's part of why it's generally good
>> practice to get more than 1 upstream if you can; it gives you more control
>> to mitigate the impact of these choices.
>>
>> It is completely possible that Baidu or CT are intentionally not
>> announcing prefixes to VZ. It is also completely possible that they are and
>> VZ is not accepting it.
>>
>>
>>
>> On Thu, Jul 21, 2022 at 11:24 AM holow29 <holow29 at gmail.com> wrote:
>>
>>> I would expect Verizon to be able to contact CT and figure out why they
>>> aren't passing the correct routes just as I might expect Baidu to do the
>>> same thing, I suppose. Ultimately, whose responsibility is it other than
>>> CT? That is my question. Maybe in this instance, it is common in the
>>> industry for this to be the responsibility of Baidu. That seems
>>> counterintuitive to me as it is Verizon without the proper route
>>> ultimately, but I am not an expert.
>>> Certainly, I think it is incorrect to ask the customer to try to resolve
>>> these issues after bringing it to the attention of these services. If
>>> Verizon couldn't reach one of Google's edge servers, would it be Verizon or
>>> Google's responsibility to fix that if the issue were an intermediate peer
>>> failing to broadcase the correct route? I have the sneaking suspicion that
>>> Verizon might actually take some action in that instance...
>>>
>>> On Thu, Jul 21, 2022 at 9:31 AM Tom Beecher <beecher at beecher.cc> wrote:
>>>
>>>> Unfortunately it seems like policy that Verizon pushes any issues that
>>>>> aren't internal routing issues to an external party, but surely they have a
>>>>> responsibility to maintain their peering and routes to external services as
>>>>> well.
>>>>>
>>>>
>>>> Baidu is behind CT. If CT is not passing on routes to 701 , what
>>>> exactly do you expect Verizon to do about it?
>>>>
>>>>
>>>> On Wed, Jul 20, 2022 at 6:40 PM holow29 <holow29 at gmail.com> wrote:
>>>>
>>>>> To follow up on this:
>>>>> I've engaged Verizon's executive office to finally try to get to a
>>>>> network engineer (because I don't have a contact myself). The (proxied)
>>>>> response from the engineer was that they aren't receiving any announcements
>>>>> for these routes to AS701, and I would need to take it up with Baidu. I
>>>>> guess I would like to understand if that seems reasonable to people since
>>>>> it is my presumption that Baidu would return and say something similar
>>>>> (that they advertise their routes to their peers correctly and to take it
>>>>> up with Verizon). To me, it seems like there is clearly a failing in one of
>>>>> Verizon's peers where they are not advertising or accepting this route
>>>>> correctly, but that it would be incumbent on Verizon to do the legwork to
>>>>> fix it since they are the ones who know their peering agreements and have
>>>>> these contacts. Unfortunately it seems like policy that Verizon pushes any
>>>>> issues that aren't internal routing issues to an external party, but surely
>>>>> they have a responsibility to maintain their peering and routes to external
>>>>> services as well. In other words, this type of buckpassing does not seem
>>>>> right to me (and I've heard it from them on other routing issues before),
>>>>> especially given that they are the ones empowered to fix it. Any thoughts?
>>>>>
>>>>> (As it happens, pan.baidu.com now resolves to an IP range that is
>>>>> routable by Verizon, but it could always revert, and it seems like Verizon
>>>>> should have these routes regardless.)
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Fri, Jun 24, 2022 at 7:41 AM Matthew Huff <mhuff at ox.com> wrote:
>>>>>
>>>>>> From my limited vantage point it appears that there is some issue
>>>>>> between Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is
>>>>>> only advertising pieces of it globally (or at least from what I can see).
>>>>>> In our tables,we are receiving none from Verizon of  the subnets that are
>>>>>> advertised directly from Baidu (origin AS of 38365). The few within that
>>>>>> registered range that have a different origin AS are coming to us from
>>>>>> Verizon. For example:
>>>>>>
>>>>>>
>>>>>>
>>>>>> *>   182.61.0.0/19    144.121.203.141                        0 46887
>>>>>> 3356 4134 58466 38365 i
>>>>>>
>>>>>> *>   182.61.0.0/18    144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.32.0/19   144.121.203.141                        0 46887
>>>>>> 3356 4134 58466 38365 i
>>>>>>
>>>>>> *>   182.61.64.0/18   204.148.121.221                        0 701
>>>>>> 6453 55967 i
>>>>>>
>>>>>> *    182.61.128.0/23  204.148.121.221                        0 701
>>>>>> 4134 4134 4134 4134 4134 58540 ?
>>>>>>
>>>>>> *>   182.61.130.0/24  144.121.203.141                        0 46887
>>>>>> 6461 4134 23724 38365 38365 38365 i
>>>>>>
>>>>>> *>   182.61.130.0/23  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.131.0/24  144.121.203.141                        0 46887
>>>>>> 6461 4134 23724 38365 38365 38365 i
>>>>>>
>>>>>> *>   182.61.132.0/23  144.121.203.141                        0 46887
>>>>>> 3356 4134 58466 38365 i
>>>>>>
>>>>>> *>   182.61.132.0/22  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.134.0/23  144.121.203.141                        0 46887
>>>>>> 3356 4134 58466 38365 i
>>>>>>
>>>>>> *>   182.61.136.0/22  144.121.203.141                        0 46887
>>>>>> 3356 4134 58466 38365 i
>>>>>>
>>>>>> *>   182.61.136.0/21  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.140.0/22  144.121.203.141                        0 46887
>>>>>> 3356 4134 58466 38365 i
>>>>>>
>>>>>> *>   182.61.144.0/21  144.121.203.141                        0 46887
>>>>>> 3356 4134 58466 38365 i
>>>>>>
>>>>>> *>   182.61.144.0/20  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.160.0/19  204.148.121.221                        0 701
>>>>>> 6453 55967 i
>>>>>>
>>>>>> *>   182.61.192.0/23  144.121.203.141                        0 46887
>>>>>> 3356 4134 58540 i
>>>>>>
>>>>>> *>   182.61.194.0/23  144.121.203.141                        0 46887
>>>>>> 3356 4134 58540 i
>>>>>>
>>>>>> *>   182.61.200.0/22  144.121.203.141                        0 46887
>>>>>> 6461 4134 23724 38365 i
>>>>>>
>>>>>> *>   182.61.200.0/21  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.216.0/21  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.223.0/24  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>> *>   182.61.224.0/19  144.121.203.141                        0 46887
>>>>>> 6461 4134 58466 38365 38365 i
>>>>>>
>>>>>>
>>>>>>
>>>>>> We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of
>>>>>> our other peers:
>>>>>>
>>>>>>
>>>>>>
>>>>>> asr-inet2#sh ip bgp 182.61.200.0/21
>>>>>>
>>>>>> BGP routing table entry for 182.61.200.0/21, version 15779018
>>>>>>
>>>>>> Paths: (2 available, best #2, table default)
>>>>>>
>>>>>>   Advertised to update-groups:
>>>>>>
>>>>>>      3
>>>>>>
>>>>>>   Refresh Epoch 1
>>>>>>
>>>>>>   54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365
>>>>>> 119.75.208.225)
>>>>>>
>>>>>>     148.77.99.201 from 148.77.99.201 (24.157.4.25)
>>>>>>
>>>>>>       Origin IGP, localpref 100, valid, external, atomic-aggregate
>>>>>>
>>>>>>       rx pathid: 0, tx pathid: 0
>>>>>>
>>>>>>       Updated on Apr 29 2022 21:02:05 EDT
>>>>>>
>>>>>>   Refresh Epoch 1
>>>>>>
>>>>>>   46887 6461 4134 58466 38365 38365, (aggregated by 38365
>>>>>> 119.75.208.225)
>>>>>>
>>>>>>     129.77.17.254 from 129.77.17.254 (129.77.40.7)
>>>>>>
>>>>>>       Origin IGP, metric 0, localpref 100, valid, internal,
>>>>>> atomic-aggregate, best
>>>>>>
>>>>>>       rx pathid: 0, tx pathid: 0x0
>>>>>>
>>>>>>       Updated on May 3 2022 04:02:50 EDT
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* NANOG <nanog-bounces+mhuff=ox.com at nanog.org> *On Behalf Of *
>>>>>> holow29
>>>>>> *Sent:* Thursday, June 23, 2022 5:49 PM
>>>>>> *To:* nanog at nanog.org
>>>>>> *Subject:* Verizon no BGP route to some of AS38365 (182.61.200.0/24)
>>>>>>
>>>>>>
>>>>>>
>>>>>> I've been trying (to no avail) for over a month now to get Verizon to
>>>>>> investigate their lack of BGP routing to 182.61.200.0/24, which
>>>>>> hosts Baidu Wangpan at  pan.baidu.com (Baidu's cloud
>>>>>> services/equivalent of Google Drive).
>>>>>>
>>>>>>
>>>>>>
>>>>>> Easily verified through Verizon's Looking Glass.
>>>>>>
>>>>>>
>>>>>>
>>>>>> We all know Verizon's BGP routing is a disaster, but does anyone have
>>>>>> any ideas?
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20220721/960853a6/attachment.html>

• Previous message (by thread): Verizon no BGP route to some of AS38365 (182.61.200.0/24)
• Next message (by thread): Verizon no BGP route to some of AS38365 (182.61.200.0/24)
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]