InfoWorld Column on Netcom-Cisco Ampersand Collapse
bob_metcalfe at infoworld.com
Tue Jul 9 15:49:02 UTC 1996
You saw that Craig Huegen has caught me in ANOTHER error.
He's of course right (way below) that:
The Network Works. No Excuses.
is Cisco's slogan, not Netcom's, as I incorrectly wrote in my current
InfoWorld column. Netcom doesn't seem to have a slogan. I stand corrected
AGAIN. A copy of my incorrect InfoWorld column is below FYI. Nothing
slips by NANOG (;->).
Not to get back at Mr. Huegen, but he should note that Cisco is not "cisco"
By the way, Mr. Huegen, the well-known fact that the Internet offers no
service guarantees has not, as you've written, escaped me. This well-known
fact is one of those we are working to FIX.
Also, tell us, what was the "original purpose" of the Internet? Not that
it matters much.
Ever your fan and loyal opposition,
/Bob Metcalfe, InfoWorld
InfoWorld, July 8, 1996 (www.infoworld.com)
Netcom-Cisco outage could foreshadow
much bigger collapses ahead
When a tropical storm grows large enough -- winds exceeding 75
mph -- we call it a hurricane and give it a name. It's time
something similar with Internet outages.
Borrowing a threshold used by our Federal Communications
Commission in the reporting of telephone outages, when more than
50,000 people are denied their Internet access for more
than an hour,
let's call it an Internet collapse and give it a name.
Let's call the
threshold a 50Kx1 collapse, or a 50 kilolapse.
Then we can say that, two weeks ago, the Internet suffered a
400Kx13 collapse, or a 5.2 megalapse. Beginning in the
June 18, the 400,000 customers of Netcom On-Line Communication
Services Inc. (http://www.netcom.com) experienced not just the
usual worsening afternoon Internet brownout but lost their
mail and Web access for 13 hours. (See "Netcom service
12-hour shutdown," June 24, page 3.)
Netcom says that this 5.2 megalapse was triggered by an engineer
incorrectly typing an ampersand into a router made by Cisco
Inc. (http://www.cisco.com). This typo was followed by "a
non-Netcom BGP [Boundary Gateway Protocol] routes being
introduced into our OSPF [open shortest path first] network
backbone. This led to a chain reaction of routing protocol
fluctuations, which in turn overloaded a majority of the gateway
routers on the Netcom WAN. Our network support staff diagnosed
the problem early and worked through the night rebuilding the
routing tables of our hub and POP routers."
So let's name this the Netcom-Cisco Ampersand Collapse.
Netcom CEO Dave Garrison apologized to his customers on KGO
talk radio in San Francisco. He explained that the collapse was
caused by human error. He admitted that the Ampersand Collapse
had overwhelmed Netcom's telephone support. He promised to meet
with Cisco, maker of most of Netcom's 100 routers, about
preventing future outages.
Interviewed by The Boston Globe, Garrison explained the
growing rapidly and there is plenty of room for
competition. Then he
said, "Internet companies face ruthless competition and
billions to spend on reliability upgrades."
Uh-oh, this despite Netcom's trademarked slogan: The Network
Works. No Excuses.
Now, to err is human, and Internet fogies ask us to accept
megalapse as nothing new, no big deal. But Garrison's upcoming
meeting with Cisco is important. Cisco should continuously
the software with which its routers are programmed so that
catastrophic human errors are less likely.
Ed Kozel, Cisco's chief technology officer, writes that "network
routing is quite susceptible to human error... complete
driving routing architecture development... in recent years
a lot of
work has gone into creating interdomain routing firewalls and
untrusted routing gateway functions, the result being that,
routing misbehavior is usually confined to a specific domain."
So we should be encouraged that the Netcom-Cisco Ampersand
Collapse did not escape Netcom and go Internetwide, this time.
While Netcom and Cisco are at it, they should find a way to make
Internet error messages more informative. Throughout the
Ampersand Collapse, Netcom customers were told that their user
names and passwords were incorrect, their calls were
network connections were lost, or nothing at all as their
session screens hung.
Now why has Netcom not offered each of its 400,000 customers a
refund for the access lost during the megalapse? Let's see,
be, say, half a day out of 30, or typically 33 cents each.
The Netcom-Cisco Ampersand Collapse and other major outages
should be prominent agenda items at upcoming meetings of
Unfortunately, my favorite of such meetings, those of the North
American Network Operators Group (NANOG), are not likely to
take systematic outage analysis seriously. As one NANOG wag put
it, "This is the 'net, people, deal with it."
What's needed is for NANOG to deal with it. Another NANOG
participant minimized the Netcom-Cisco 5.2 megalapse with this
arithmetic: Since the Internet has 60 million users, the
inconvenienced far fewer than 1 percent -- some collapse.
He has a
point. There is ample room for much bigger Internet
maybe eventually some gigalapses. (See what else the NANOG wags
are writing about at
Bob Metcalfe invented Ethernet in 1973 and founded 3Com Corp. in
1979. He receives E-mail at bob_metcalfe at infoworld.com via the
Copyright © 1996 by InfoWorld Publishing Company
At 4:15 PM 7/8/96, Craig A. Huegen wrote:
>Received: by ccmail from lserver.infoworld.com
>>From c-huegen at quad.quadrunner.com
>X-Envelope-From: c-huegen at quad.quadrunner.com
>Received: from quad.quadrunner.com by lserver.infoworld.com with smtp
> (Smail126.96.36.199 #12) id m0udR8U-000x1xC; Mon, 8 Jul 96 17:58 PDT
>Received: from localhost (c-huegen at localhost) by quad.quadrunner.com
>(8.7.5/8.7-quad) with SMTP id QAA12003; Mon, 8 Jul 1996 16:14:36 -0700
>Date: Mon, 8 Jul 1996 16:14:36 -0700 (PDT)
>From: "Craig A. Huegen" <c-huegen at quad.quadrunner.com>
>To: Michael Dillon <michael at memra.com>
>cc: nanog at merit.edu, bob_metcalfe at infoworld.com
>Subject: Re: Hurricanes redefined!
>In-Reply-To: <Pine.BSI.3.93.960708104725.22916I-100000 at sidhe.memra.com>
>Message-ID: <Pine.QUAD.3.94.960708155947.11988A-100000 at quad.quadrunner.com>
>Content-Type: TEXT/PLAIN; charset=US-ASCII
>On Mon, 8 Jul 1996, Michael Dillon wrote:
>==>The official definition of a hurricane is winds in excess of 5 km/hr
>==>for a duration of at least 1 hour. Read all about it at
>Interesting as well is Bob's incorrectness once again:
> "Uh-oh, this despite Netcom's
> trademarked slogan: The Network
> Works. No Excuses. "
>See http://www.cisco.com/public/copyright.html, in which you'll find:
>"All rights reserved. No portion of this service may be reproduced in any
>form, or by any means, without prior written permission from Cisco
>Systems, Inc. [...], The Network Works. No Excuses. are service marks;
>[...] of Cisco Systems, Inc.[...]"
>Bob also states that:
> "Cisco should
> continuously improve the software with
> which its routers are programmed so
> that catastrophic human errors are less
> likely. "
>Which gives the connotation that cisco Systems doesn't constantly improve
>IOS; everyone who's worked with cisco's software knows it's constantly
>Bob also states the following:
> "Unfortunately, my favorite of such
> meetings, those of the North American
> Network Operators Group (NANOG),
> are not likely to take systematic outage
> analysis seriously. As one NANOG
> wag put it, "This is the 'net, people, deal
> with it.""
>What Bob fails to mention is that no one has a service-level agreement
>with the Internet. The Internet is designed this way--my network connects
>to your network. It is _NOT_ under control of one body. It's very hard
>to GUARANTEE outages to _anyone_ without monetary value involved. And
>generally, "my networks connects to your network" does not have enough
>monetary value to warrant SLA contracts of service. Bob, I challenge you
>to find an Internet Service Provider that gives an END-TO-END service
>level agreement for the Internet. That is, if my web site isn't fast
>enough, you have escalation/remedy procedures. If Joe Blow's sendmail has
>crapped out, you have escalation/remedy procedures. I'll buy you dinner
>if you find one.
>Bob once again forgets the original purpose behind the Internet, and he
>apparently has permanently doffed his engineer hat for his non-technical
>businessperson hat long ago.
Dr. Robert M. ("Bob") Metcalfe
Executive Correspondent, InfoWorld and
VP Technology, International Data Group
Internet Messages: bob_metcalfe at infoworld.com
Voice Messages: 617-534-1215
Conference Chairman for
ACM97: The Next 50 Years of Computing
San Jose Convention Center
March 1-5, 1997
More information about the NANOG