Microsoft Express Routes woes...

Wed Jun 27 17:57:44 UTC 2018

Hi Craig,

Microsoft apologizes for your routing issue.  Our Wide Area Networking and
Express Route teams were already aware of this issue from your ticket.  I
have communicated with their leadership, and they are working on a fix for
you.

Please feel free to contact me at ingrid.erkman at microsoft.com if you would
like to follow up off list.

Thanks,

Ingrid Erkman

Director of Interconnection

Microsoft

On Wed, Jun 27, 2018 at 8:58 AM, Craig <cvuljanic at gmail.com> wrote:

> This is classic.......
>
> We have a direct BGP peering session to Microsoft using Express Routes for
> the public peering session for services like Email, one drive, etc. this
> uses MS Public. We also have/use MS Azure Public as well as MS Azure
> Private in place for a few years now. We have had this happen a few times
> already where one team at MS makes a routing change, and the other team is
> either not aware of the change, or else doesn't communicate the change
> properly.
>
> So late last night, while a change was being made to a completely different
> area of our network, they asked that I back out my change due to our entire
> organization not being able to access share-point online, or MS One drive.
>
> I had zero evidence it was our change. further investigation on our border
> routers, revealed all four (4) of our ISP's were advertising the MS block
> as a /24 prefix:
>
> A:MY_NAME_CHANGED# show router 1053 bgp routes 13.107.136.0/24
> ========================================================================
>  BGP Router ID:10.11.0.29       AS:122         Local AS:122
> ========================================================================
>  Legend -
>  Status codes  : u - used, s - suppressed, h - history, d - decayed, * -
> valid
>                  l - leaked, x - stale, > - best, b - backup, p - purge
>  Origin codes  : i - IGP, e - EGP, ? - incomplete
>
> ========================================================================
> BGP IPv4 Routes
> ========================================================================
> Flag  Network                                            LocalPref   MED
>       Nexthop (Router)                                   Path-Id     Label
>       As-Path
> ------------------------------------------------------------
> -------------------
> u*>?     13.107.136.0/24                                    250
> 10450
>       4.49.118.153                                       None        -
>       3356 8075 8068
> ------------------------------------------------------------
> -------------------
> Routes : 1
>
> HOWEVER MS Express Routes was advertising this:
>
> A:MY_NAME_CHANGED# show router 1053 bgp routes 13.107.136.0/22
> ========================================================================
>  BGP Router ID:10.11.0.29       AS:122         Local AS:122
> ========================================================================
>  Legend -
>  Status codes  : u - used, s - suppressed, h - history, d - decayed, * -
> valid
>                  l - leaked, x - stale, > - best, b - backup, p - purge
>  Origin codes  : i - IGP, e - EGP, ? - incomplete
>
> ========================================================================
> BGP IPv4 Routes
> ========================================================================
> Flag  Network                                            LocalPref   MED
>       Nexthop (Router)                                   Path-Id     Label
>       As-Path
> ------------------------------------------------------------
> -------------------
> i  13.107.136.0/22                                    300         0
>       157.229.11.66                                      None        -
>       12076
>
> So another call to MS support and escalations to get this more specific
> prefix fixed or if Express routes can advertise this more specific vs the
> /22 block.
>
> *Questions:*
> Has anyone else ran into this with MS if you have a direct peering session
> to them?
> Has anyone did an audit on the route table's received from MS over express
> routes, vs what they receive from their ISP's and noticed the differences?
>
> MS needs to seriously think about:
> Careful coordination of routing changes
> Policies to prevent specific routes being advertised while larger blocks
> are advertised over express routes.
>
> Anyway I am tired, as I have not had much sleep, any comments on this,
> would like to hear from you.
>
>
>
> -Craig
>