CloudFlare issues?

Mon Jun 24 14:11:22 UTC 2019

On Mon, Jun 24, 2019 at 08:18:27AM -0400, Tom Paseka via NANOG wrote:
> a Verizon downstream BGP customer is leaking the full table, and some more
> specific from us and many other providers.

It appears that one of the implicated ASNs, AS 33154 "DQE Communications
LLC" is listed as customer on Noction's website:
https://www.noction.com/clients/dqe

I suspect AS 33154's customer AS 396531 turned up a new circuit with
Verizon, but didn't have routing policies to prevent sending routes from
33154 to 701 and vice versa, or their router didn't have support for RFC
8212.

I'd like to point everyone to an op-ed I wrote on the topic of "BGP
optimizers": https://seclists.org/nanog/2017/Aug/318

So in summary, I believe the following happened:

    - 33154 generated fake more-specifics, which are not visible in the DFZ
    - 33154 announces those fake more-specifics to at least one customer (396531)
    - this customer (396531) propagated to to another upstream provider (701)
    - it appears that 701 did not sufficient prefix filtering, or a maximum-prefix limit

While it is easy to point at the alleged BGP optimizer as the root
cause, I do think we now have observed a cascading catastrophic failure
both in process and technologies. Here are some recommendations that all
of us can apply, that may have helped dampen the negative effects:

    - deploy RPKI based BGP Origin validation (with invalid == reject)
    - apply maximum prefix limits on all EBGP sessions
    - ask your router vendor to comply with RFC 8212 ('default deny')
    - turn off your 'BGP optimizers'

I suspect we, collectively, suffered significant financial damage in
this incident.

Kind regards,

Job