BBN outage

Sun Oct 13 04:06:46 UTC 1996

>From: Per Gregers Bilse <bilse at EU.net>
> > Because loss of connectivity for a day costs 10's of thousands
> > of dollars and, evidently, even the "big" companies are not
> > immune from "acts of god" like what happened today.
> 
> Please distinguish between singly-connected and singly-homed, they
> really are two different things.  If Sun has just a plain, regular
> connection, a garden variety leased line outage will take them out.
> If OTOH they had had a connection to one of BBN's other CA POPs, they
> would have been just fine.  It is intuitively obvious that the
> advantage of multihoming does not outweigh the additional complexity,
> cost and failure modes, whereas multiconnecting can be done at
> reasonable cost while actually providing more reliable service
> overall.  All information clearly indicates that the PA outage was
> one of these near-enough disasters that can and do happen, just like
> hurricanes and floods and whatever.  You can't blame BBN for that,
> but you could conceivably blame Sun, InfoWorld and/or whoever for
> having just a single connection, if they're supposed to take their
> Internet business seriously -- do these people really want to be
> taken out by a blown CSU/DSU?

I did not say multi-homing by itself would be a cure-all.
I picked Sun as an example among the affected customers
because they do have several interconnected physical
locations in the bay area, US-wide and abroad and also
in-house expertise on routing. It is just that a whole
bunch of smart people running Sun's NOC must have felt
helpless.. If only they had a pipe going out to somewhere
else, another provider in the bay area in this case, they
could have reconfigured routers and minimized the hit.

In hindsight, solutions to tolerate a given failure are
very easy to come by but shouldn't be talked down. I did not
blame BBN--shit happens--but I wouldn't assume that their
NOC is necessarily designed to be more fault tolerant
than what a smaller ISP can or what the people at Sun
could if they didn't somehow assume IP transit out of
BBN was fault tolerant enough. For all you know, Sun
might have multiple pipes connecting to BBN, multipe
routers and CSU/DSU spares. I know for sure a client
of BBN's cupertino POP was affected because
all BBN traffic in the bay area, as far as I can tell,
went/goes through the stanford location.

Sanjay.