off-topic: summary on Internet traffic growth History

John Lee john at internetassociatesllc.com
Wed Aug 11 18:13:47 UTC 2010


Andrew,

Earlier this week I had a meeting with the ex-Director of the Network Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to handle the growth in traffic including these new facilities called peering points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they saw traffic on their switches grow at 50-70% growth every 6 months. By the last half of 1996 there was a head of line blocking problem on the DEC FDDI switches that was "regularly" bringing down the Internet. The architecture had lower traffic circuits were going through concentrators while higher traffic circuits were directly attached to ports on the switchs.



MFS-Datanet was not going to take the hit for the interruptions to the Internet and was going to inform the trade press there was a problem with DEC FDDI switches so Digital "gave" six switches for the re-architecture of the MAEs to solve the problem. Once this problem was solved the first quarter of 1997 saw a 70% jump in traffic that quarter alone. This "historical event" would in my memory be the genesis of the 100% traffic growth in 100 days legend. (So it was only 70% in 90 days which for the marketing folks does not cut it so 100% in 100 days sounds much better?? :) )



MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.



Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve whose team produced the TNT line of modem/ISDN to Ethernet central site concentrators (in the early ninties) that drove a large portion of the user traffic to the Internet at the time, generating the "bubble".



John (ISDN) Lee
________________________________________
From: Andrew Odlyzko [odlyzko at umn.edu]
Sent: Wednesday, August 11, 2010 12:55 PM
To: nanog at nanog.org
Subject: off-topic: summary on Internet traffic growth myths

Since several members of this list requested it, here is a summary
of the responses to my request for information about Internet growth
during the telecom bubble, in particular the perceptions of the
O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days"
myth.

First of all, many thanks to all those who responded, on and off-list.
This involved extensive correspondence and some long phone conversations,
and helped fill out the picture of those very confusing times (and
also made it even clearer than before that there were many different
perspectives on what was happening).

The entire message is rather long, but it is written in sections,
to make it easy to get the gist quickly and neglect the rest.

Andrew


-------------------------------------------------------------------

1.  Short summary: People who got into the game late, or had been
working at small ISPs or other enterprises, were generally willing
to give serious credence to the "Internet doubling every 100 days"
tale.  The old-timers, especially those who worked for large ISPs
or other large corporate establishment or research networks, were
convinced by the late 1990s that this tale was false, but did not
talk about it publicly, even inside the NANOG community.

-------------------------------------------------------------------

2.  Longer version: The range of views was very wide, and hard to
give justice to in full.  But there seemed to be two distinct
groups, and the consensus views (which obviously exclude quite
a few people) appear to have been:

2A: Those who entered the field in the late 1990s, especially
if they worked for small ISPs or other small enterprises, tended
to regard the claim seriously.  (But it should be remarked that
hardly anybody devoted too much effort or thought to the claim,
they were too busy putting out fires in their own backyards to
worry about global issues.)  They remembered periods of desperate
efforts to keep up with exploding demand in their businesses.
We saw just a few hours ago a post about LINX growing 5.5x in
one year.  Somebody else wrote about growing their business's
traffic 1,000x in 2 years, or about 30x per year.  People involved
in such incidents often tended to think that their experience
during such times might not have been untypical.

2B: Those who worked at places with large traffic, and especially
those who got into the field in the early 1990s, were quite sure
by the late 1990s that the UUNet fable was just that.  Comments
regarding everything emanating from UUNet during that period
included phrases like "blowing smoke," "rolling our eyes," "taking
it with a rock of salt."  They had no direct knowledge of what
went on inside UUNet, but from watching peering traffic, talking
to salespeople about customer losses and wins, and to suppliers
about deliveries of equipment, they could be pretty certain that
neither the traffic nor the capacity of UUNet could be exploding
at the mythical rates.  They could see occasional spikes in
traffic growth at some customers, or in some parts of their
networks, but overall could see traffic growth settling down
to a fairly regular doubling or a bit more than doubling each
year.

However, they did not discuss this in public, and I discuss that
below, in point #4.

-------------------------------------------------------------------

3.  Growth spurt in mid-1990s: The old-timers also provided very
informative feedback about the global Internet traffic growth
spurt in the 1995-96 time frame.  It did hit suddenly and
unexpectedly.  (I am still trying to get confirmation on this,
but I believe one of the informants said that before this
period, the engineers at that person's Tier-1 ISP would routinely
double the forecasts provided by the marketing team.  At the peak
of the bubble, the marketers would demand that the engineers plan
for double the capacity that the engineers thought was going
to be necessary.)  Moreover, the dramatic slowdown in traffic
growth that took place in 1997 was, at least in a number of
cases, due substantially to a capacity crunch.  Router and
photonic equipment manufacturers did not have the technology
needed for the traffic, and ILECs were slow in supplying
access as well as backbone links.  Hence the traffic growth
spurt in 1996-96 was followed by a capacity growth spurt
in 1997-98, which helped provide more credibility for the
myth.

-------------------------------------------------------------------

4.  Information viscosity: This incident provides far more
information confirming the concept of "information viscosity"
that I wrote about in my paper, that important and relevant
information was available, but was not widely dispersed.

Why did the information that Internet traffic was not doubling
every 100 days get out to the public?  It was not a closely
guarded national security secret, after all.

There seemed to be many reasons operating.  In the case of
Genuity (as the quote from Scott Marcus in my paper explains)
it was a high-level decision, that going public would hurt
the company, as it might be suspected of losing market share.
But that seemed to be unusual, in that Genuity, while a
major Tier-1 ISP, was small and independent, and so had
people like Scott who were engineers, yet involved in
policy making.  At other places, other dynamics operated.

At AT&T, for example, I was called in to a meeting with
the management of WorldNet, the AT&T ISP unit, at the
end of 2000.  They had been telling their customers that
Internet traffic was growing 10x per year, and some of
those customers asked them about the discrepancy between
that claim and the estimates of some folks from AT&T
Labs - Research (Kerry Coffman and myself) that growth
was just 2x per year.  Now it is an interesting perspective
on "information viscosity" and AT&T (and other large
bureaucratic organizations) that those folks had not
heard of Kerry's and my work, even though we had publicized
it at the company.  But in any event, at that meeting,
I did succeed in convincing them that Internet traffic
was growing only 2x per year (using evidence in my papers
with Kerry, as well as extensive additional data from
within AT&T itself), and they agreed they would not
propagate the myth among customers.  But the interesting
thing was that many of the attendees (who included quite
a few engineering types, not just the management) seemed
disappointed.  That really surprised me.  After all,
AT&T's Internet traffic was growing just about 4x per
year, which meant our market share was doubling, instead
of being cut in half.  But it seems that many of them
had really bought into the Internet dream.  And, furthermore,
the story that we were losing market share was a good one
to pry additional resources from the corporation.

Several of the NANOG old-timers said that they felt constrained
from speaking about the falsity of the myth of astronomical
growth rates by group solidarity.  After all, it was a
small, select group, and bad-mouthing one of their own
in front of outsiders or even NANOG newbies did not seem
the right thing to do.  Then there was the additional
factor that one person discussed very explicitly, and
that I infer also applied in other cases.  The myth was
useful.  Back in the mid-1990s, when it was not a myth,
it did spur equipment suppliers and ILECs to greater
effort.  And afterwards, it was handy in internal fights
over power and resources.  If the Internet was exploding,
one should not worry about closely examining expansion
plans.

In some ways, the persistence of false perceptions about
Internet traffic growth mirrors that of utilization rates
of data networks.  When I wrote the paper "Data networks
are lightly utilized, and will stay that way," back in 1998,

     http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf

the more clueful data network engineers all knew that
utilization rates were generally low, and usually had
a good understanding as to why.  But top management, as
well as the research community, were almost uniformly
convinced that congestion was the rule.  It took me a
while to understand the dynamics of the situation, in
which network engineers and managers found it easier
to say that they were experiencing 80% utilization and
20% packet losses and needed to upgrade, without having
to explain to CEOs, CIOs, and especially to clueless
CFOs, that this referred just to the peak hour on one crucial
corporate WAN link, and yet justified a big new investment.
Few people were lying, but many had incentives to maintain
delusions among the top levels of the hierarchy.

These demonstrations of information viscosity of course
undermine the foundations of much of modern economics,
especially of the efficient markets hypothesis.  But that
last myth is even more durable than "Internet traffic
doubling every 100 days," especially since it does not
lead directly to lots of dark fiber lying around unused,
and lots of companies bankrupt.  Information viscosity
in facts and ideas in economics is very high, so we
should not expect any changes on that scene.


-------------------------------------------------------------------

5.  As a reminder, the paper that led to this discussion is
"Bubbles, gullibility, and other challenges for economics,
psychology, sociology, and information sciences,"

     http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf

The page with source materials from the bubbles times,

    http://www.dtc.umn.edu/~odlyzko/isources/

now has, in addition to a transcript of the O'Dell lecture
at Stanford in May 2000, a copy of the Sidgmore paper from
the Vortex98 Conference of May 1998.



More information about the NANOG mailing list