<html><head>
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
</head><body style="font-family: tt;" bgcolor="#FFFFFF" text="#000000"><div
style="font-family: tt;"><span style="font-family: monospace;">This is
what looked happened:<br><br>There was a large scale BGP 'leak' incident
causing about 20k prefixes for 2400 network (ASNs) to be rerouted
through AS396531 (a steel plant) and then on to its transit provider:
Verizon (AS701) Start time: 10:34:21 (UTC) End time: 12:37 (UTC)<br>All
ASpaths had the following in common:<br>701 396531 33154<br><br><br>33154
(DQECOM ) is an ISP providing transit to 396531.<br>396531 is by the
looks of it a steel plant. dual homed to 701 and 33154. <br>701 is
verizon and accepted by the looks of it all BGP announcements from
396531<br><br>What appears to have happened is that 33154 those routes
were propagated to 396531, which then send them to Verizon and voila...
there is the full leak at work.<br>(DQECOM runs a BGP optimizer
(<a class="moz-txt-link-freetext" href="https://www.noction.com/clients/dqe">https://www.noction.com/clients/dqe</a> , thanks Job for pointing that out,
more below)<br><br>As a result traffic for 20k prefixes or so was now
rerouted through verizon and 396531 (the steel plant)<br><br>We've seen
numerous incidents like this in the past<br>lessons learned:<br>1) if
you do use a BGP optimizer, please FILTER!<br>2) Verizon... filter your
customers, please!<br><br><br>Since the BGP optimizer introduces new
more specific routes, a lot of traffic for high traffic destinations
would have been rerouted through that path, which would have been
congested, causing the outages.<br>There were many cloudflare prefixes
affected, but also folks like Amazon, Akamai, Facebook, Apple, Linode
etc.<br><br>here's one example for Amazon - CloudFront : 52.84.32.0/22.
Normally announced as a 52.84.32.0/21 but during the incident as a /22
(remember more specifics always win)<br><a class="moz-txt-link-freetext" href="https://stat.ripe.net/52.84.32.0%2F22#tabId=routing&routing_bgplay.ignoreReannouncements=false&routing_bgplay.resource=52.84.32.0/22&routing_bgplay.starttime=1561337999&routing_bgplay.endtime=1561377599&routing_bgplay.rrcs=0,1,2,5,6,7,10,11,13,14,15,16,18,20&routing_bgplay.instant=null&routing_bgplay.type=bgp">https://stat.ripe.net/52.84.32.0%2F22#tabId=routing&routing_bgplay.ignoreReannouncements=false&routing_bgplay.resource=52.84.32.0/22&routing_bgplay.starttime=1561337999&routing_bgplay.endtime=1561377599&routing_bgplay.rrcs=0,1,2,5,6,7,10,11,13,14,15,16,18,20&routing_bgplay.instant=null&routing_bgplay.type=bgp</a><br><br>RPKI
would have worked here (assuming you're strict with the max length)!<br><br><br>Cheers<br> Andree<br><br></span><br><span>My
secret spy satellite informs me that Dmitry Sherman wrote On
2019-06-24, 3:55 AM:</span><br><blockquote
cite="mid:B646241725B36940945DFA68339668E93ADA00A0@exchange.exchange.interhost.co.il"
type="cite"><pre wrap="">Hello are there any issues with CloudFlare services now?
Dmitry Sherman
<a class="moz-txt-link-abbreviated" href="mailto:dmitry@interhost.net">dmitry@interhost.net</a>
Interhost Networks Ltd
Web: <a class="moz-txt-link-freetext" href="http://www.interhost.co.il">http://www.interhost.co.il</a>
fb: <a class="moz-txt-link-freetext" href="https://www.facebook.com/InterhostIL">https://www.facebook.com/InterhostIL</a>
Office: (+972)-(0)74-7029881 Fax: (+972)-(0)53-7976157
</pre></blockquote><br></div></body></html>