Fiber cut in SF area

George William Herbert gherbert at
Mon Apr 13 19:32:49 CDT 2009

Matthew Petach writes:
>"protected rings" are a technology of the past.  Don't count on your
>vendor to provide "redundancy" for you.  Get two unprotected runs
>for half the cost each, from two different providers, and verify the
>path separation and diversity yourself with GIS data from the two
>providers; handle the failover yourself.  That way, you *know* what
>your risks and potential impact scenarios are.  It adds a bit of
>initial planning overhead, but in the long run, it generally costs a
>similar amount for two unprotected runs as it does to get a
>protected run, and you can plan your survival scenarios *much*
>better, including surviving things like one provider going under,
>work stoppages at one provider, etc.

This completely ignores the grooming problem.

About five years ago, we had a major WebEx outage caused by
our diverse path routed fibers both being groomed into the
same new cable / new path.

We had the contracts.  We paid the money.  We got the data.
We got updates to the data.  The updates said we were still
fine and all good.

The new data lied.  Downtown SJ backhoe hit damaged the cable,
and took down 1 of our 2 links.  As nobody was sure what was
in it they failed to notify us that they were about to
chop the rest of it to repair the bundle.  So, about an hour
after we lost the first leg, we went dark, and there was no
coming back until the splices were all done.

(typically, while the whole operations team was out at an
offisite teambuilding effort.  pagers go beep beep beep,
and everyone hops back in the cars...)

We ran it up the flagpole to CEO level of the fiber vendor
(aggregator) and fiber physical plant owner (big 4 ISP),
as we were paying $$$ for bandwidth and were a Highly
Visible Client, and were told that they'd been making
a best effort and couldn't guarantee any better in the
future, no matter how much we paid or who we sued.

They were very apologetic, but insisted that best effort
means just that.

The only way to be sure?  Own your own fiber.  Use a microwave
link backup.

You have to get out of the game the fiber owners are playing.
They can't even keep score for themselves, much less accurately
for the rest of us.  If you count on them playing fair or
right, they're going to break your heart and your business.

-george william herbert
gherbert at

More information about the NANOG mailing list