[EXTERNAL] Re: Famous operational issues

Tue Feb 16 22:51:18 UTC 2021

There was the outage in 2014 when we got to 512K routes.  http://www.bgpmon.net/what-caused-todays-internet-hiccup/

On 2/16/21, 1:04 PM, "NANOG on behalf of Job Snijders via NANOG" <nanog-bounces+rich.compton=charter.com at nanog.org on behalf of nanog at nanog.org> wrote:

    CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.

    On Tue, Feb 16, 2021 at 01:37:35PM -0600, John Kristoff wrote:
    > I'd like to start a thread about the most famous and widespread Internet
    > operational issues, outages or implementation incompatibilities you
    > have seen.
    > 
    > Which examples would make up your top three?

    This was a fantastic outage, one could really feel the tremors into the
    far corners of the BGP default-free zone:

    https://labs.ripe.net/Members/erik/ripe-ncc-and-duke-university-bgp-experiment/

    The experiment triggered a bug in some Cisco router models: affected
    Ciscos would corrupt this specific BGP announcement ** ON OUTBOUND **.
    Any peers of such Ciscos receiving this BGP update, would (according to
    then current RFCs) consider the BGP UPDATE corrupted, and would
    subsequently tear down the BGP sessions with the Ciscos. Because the
    corruption was not detected by the Ciscos themselves, whenever the
    sessions would come back online again they'd reannounce the corrupted
    update, causing a session tear down. Bounce ... Bounce ... Bounce ... at
    global scale in both IBGP and EBGP! :-)

    Luckily the industry took these, and many other lessons to heart: in
    2015 the IETF published RFC 7606 ("Revised Error Handling for BGP UPDATE
    Messages") which specifices far more robust behaviour for BGP speakers.

    Kind regards,

    Job

E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.