CloudFlare issues?

Mon Jun 24 14:25:16 UTC 2019

This is what looked happened:

There was a large scale BGP 'leak' incident causing about 20k prefixes
for 2400 network (ASNs) to be rerouted through AS396531 (a steel plant)
and then on to its transit provider: Verizon (AS701) Start time:
10:34:21 (UTC) End time: 12:37  (UTC)
All ASpaths had the following in common:
701 396531 33154

33154 (DQECOM ) is an ISP providing transit to 396531.
396531 is by the looks of it a steel plant. dual homed to 701 and 33154.
701 is verizon and accepted by the looks of it all BGP announcements
from 396531

What appears to have happened is that 33154  those routes were
propagated to 396531, which then send them to Verizon and voila... there
is the full leak at work.
(DQECOM  runs a BGP optimizer (https://www.noction.com/clients/dqe ,
thanks Job for pointing that out, more below)

As a result traffic for 20k prefixes or so was now rerouted through
verizon and 396531 (the steel plant)

We've seen numerous incidents like this in the past
lessons learned:
1) if you do use a BGP optimizer, please FILTER!
2) Verizon... filter your customers, please!

Since the BGP optimizer introduces new more specific routes, a lot of
traffic for high traffic destinations would have been rerouted through
that path, which would have been congested, causing the outages.
There were many cloudflare prefixes affected, but also folks like
Amazon, Akamai, Facebook, Apple, Linode etc.

here's one example for Amazon - CloudFront : 52.84.32.0/22. Normally
announced as a 52.84.32.0/21 but during the incident as a /22 (remember
more specifics always win)
https://stat.ripe.net/52.84.32.0%2F22#tabId=routing&routing_bgplay.ignoreReannouncements=false&routing_bgplay.resource=52.84.32.0/22&routing_bgplay.starttime=1561337999&routing_bgplay.endtime=1561377599&routing_bgplay.rrcs=0,1,2,5,6,7,10,11,13,14,15,16,18,20&routing_bgplay.instant=null&routing_bgplay.type=bgp

RPKI would have worked here (assuming you're strict with the max length)!

Cheers
 Andree

My secret spy satellite informs me that Dmitry Sherman wrote On
2019-06-24, 3:55 AM:
> Hello are there any issues with CloudFlare services now?
>
> Dmitry Sherman
> dmitry at interhost.net
> Interhost Networks Ltd
> Web: http://www.interhost.co.il
> fb: https://www.facebook.com/InterhostIL
> Office: (+972)-(0)74-7029881 Fax: (+972)-(0)53-7976157
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20190624/7c285970/attachment.html>