Level3 worldwide emergency upgrade?

Wed Feb 6 13:50:03 UTC 2013

> Yeah, perhaps not as elegantly worded as I would have hoped, but there are
> many reasons things "go down".  Just one of those elements is the internet
> part, there's also transport, power, and other elements that combine to
> make this complex system called the internet.  If you N+N or N+1 your
> power, perhaps something similar for your connectivity is important.  Or you
> just plan to be down/broken periodically for 30 minutes and have a plan to
> cover that.

Agreed.

> The building where our NOC is located sometimes gets evacuated.  Having a
> plan for that is important.  During one visit, there was a small fire in the
> building (or so we were told).  Certainly an unexpected event that disrupted
> us for ~30 minutes.

And, if it is important to you, you will have N+N NOC's - ie, more than one, and different buildings, cities, or countries, depending on your requirement.

> The handling and response of these events certainly is important.  I do want
> to understand why and how it's so bad so if there are things as a SP in the
> community we can improve upon we can do that.

I suspect, as I touched previously, the most noise will come from the people who are the least realistic, and least prepared. Personally, I live with the expectation that whatever it is (power, fiber, transport, ISP, highways, fuel delivery, etc.) will at some point be broken, degraded, or otherwise unavailable, and you have to plan accordingly. 

Personally (and I speak for NAC) I/we don't care, really, if any upstream IP provider breaks; we have made appropriate plans to work around that in an automated fashion. Hope that answers your more general question.