Centurylink having a bad morning?

Sun Aug 30 22:53:00 UTC 2020

This is what happens when the design of 'god power' automation tools
doesn't take into account the concept of blast radius. It might be more
inconvenient to internally partition automated change management systems,
but it can also limit the effect of automation tools gone awry.

https://www.ibm.com/garage/method/practices/manage/practice_limited_blast_radius/

https://principlesofchaos.org/

On Sun, Aug 30, 2020 at 10:09 AM Baldur Norddahl <baldur.norddahl at gmail.com>
wrote:

> An outage is what it is. I am not worried about outages. We have multiple
> transits to deal with that.
>
> It is the keep announcing prefixes after withdrawal from peers and
> customers that is the huge problem here. That is killing all the effort and
> money I put into having redundancy. It is sabotage of my network after I
> cut the ties. I do not want to be a customer at an outlet who has a system
> that will do that. Luckily we do not currently have a contract and now they
> will have to convince me it is safe for me to make a contract with them. If
> that is impossible I guess I won't be getting a contract with them.
>
> But I disagree in that it would be impossible. They need to make a good
> report telling exactly what went wrong and how they changed the design, so
> something like this can not happen again. The basic design of BGP is such
> that this should not happen easily if at all. They did something unwise.
> Did they make a route reflector based on a database or something?
>
> Regards,
>
> Baldur
>
> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho <mikebolitho at gmail.com>
> wrote:
>
>> Exactly. And asking that they somehow prove this won't happen again is
>> impossible.
>>
>> - Mike Bolitho
>>
>> On Sun, Aug 30, 2020, 8:10 AM Drew Weaver <drew.weaver at thenap.com> wrote:
>>
>>> I’m not defending them but I am sure it isn’t intentional.
>>>
>>>
>>>
>>> *From:* NANOG <nanog-bounces+drew.weaver=thenap.com at nanog.org> *On
>>> Behalf Of *Baldur Norddahl
>>> *Sent:* Sunday, August 30, 2020 9:28 AM
>>> *To:* nanog at nanog.org
>>> *Subject:* Re: Centurylink having a bad morning?
>>>
>>>
>>>
>>> How is that acceptable behaviour? I shall remember never to make a
>>> contract with these guys until they can prove that they won't advertise my
>>> prefixes after I pull them. Under any circumstances.
>>>
>>>
>>>
>>> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins <
>>> joe at breathe-underwater.com>:
>>>
>>> Finally got through on their support line and spoke to level1. The only
>>> thing the tech could say was it was an issue with BGP route reflectors and
>>> it started about 3am(pacific). They were still trying to isolate the issue.
>>> I've tried failing over my circuits and no go, the traffic just dies as L3
>>> won't stop advertising my routes.
>>>
>>>
>>>
>>> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG <nanog at nanog.org>
>>> wrote:
>>>
>>> Hello,
>>>
>>>
>>>
>>> Woke up this morning to a bunch of reports of issues with connectivity
>>> had to shut down some Level3/CTL connections to get it to return to normal.
>>>
>>>
>>>
>>> As of right now their support portal won’t load:
>>> https://www.centurylink.com/business/login/
>>>
>>>
>>>
>>> Just wondering what others are seeing.
>>>
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20200830/abe3343a/attachment.html>