Conflicts and fiber cuts

Mon Mar 7 10:00:17 UTC 2022

On Sunday, March 6th, 2022 at 23:40, Sean Donelan <sean at donelan.com> wrote:

> Historically, the largest telecommunication outages have been due > to operator error

Yeah, tell me about it.

In the very recent past there was a certain Tier 1 operator who decided to move a core router between X & Y within the same datacentre.  As you might imagine, this impacted a good few hundred fibres.

Physical move happened without issue, router powered up fine, but a large subset of fibres refused to come up.

Cue escalation, lots of testing, lots of talking to vendors etc.

Eventually 72 hours later the problem was solved.

Cause ?  SFP signals too high at new location, attenuators fitted, everything came up.