Seeing Double

Thu Nov 23 05:53:11 UTC 1995

Scott,

>  	There is an often quoted statistic of the Internet
> doubling every 12 months.  If we look at a snippet of 
[...]
> I've see a few guesses on this list as to a doubling rate, but
> I'm wondering if there is a way to judge this growth in some 
> sort of external and non-proprietary way.  
>
> Perhaps a relationship between "average traffic" and the number
> of routes?  This probably wouldn't hold in the specific
> (because of the degree of use of CIDR at a particular ISP), 
> but may hold in the aggregate. Thinking caps? 

It used to be that there were three ways to measure the growth of the
Internet, these being the growth in the monthly total NSFnet backbone
traffic, the size of a full forwarding table (or the size of the Merit
PRDB, which was related to the size of a full forwarding table by a
fairly constant factor), and the number of hosts discovered by the
DNS host count.  These all grew in near-enough lockstep, with doubling
times consistently in the range of 10 to 15 months over a period of
many years, that one could come to the conclusion that they were
actually telling you something about the global whole, even though
any of those measurements by itself might have been less convincing.

The deployment of BGP4 pretty much screwed up the size of a full
forwarding table as a measure of anything of greater significance than
the size of a full forwarding table (which, of course, now also varies
in size and content depending on where you measure it).  While the
jury is still out on whether we've actually caused a fundamental change
to the way in which global routing information grows, or whether we're
just in transition to a shallower curve, where the growth in global
routing information will again match the growth rate of the Internet as
a whole but with a smaller constant of proportionality, it is almost
certainly the case that the last couple of years of data won't tell you
much of anything at all.

This leaves the DNS host counts.  I haven't seen any results from these
for a long time now (are they still done?), but in any case this was
always the least satisfactory measure of the three because of the large
number of things you could list as being wrong with them.  These were
included in plots I saw mostly because they happened to often match the
other two growth rates, if they hadn't I suspect they would have been
discarded without further concern.

You could certainly measure your own traffic growth rate, but to
know if this represented a sample consistent with the global growth rate
you'd need other peoples' data to compare it to, and since the latter
comparison could be construed by the simple minded (including marketing
people and the press, no doubt) to have connotations with respect to "market
share" no one in their right mind is going to want to participate in an
I'll-show-you-mine-if-you-show-me-your's competitive rathole like this.
You could also try using measurements of NAP traffic to characterize the
global whole, but in fact without other data to correlate this with you've
still got to wonder if this is representative of anything more than
just the growth rate of NAP traffic.

I personally think we've sort of seen the end of a golden age of
Internet building, and that hope of maintaining any sort of macro-level
sense of the scale and nature of the whole thing being built (rather than
just guarded views of individually-owned chunks of it) is diminishing.  I
do find it exceedingly annoying that all sorts of bits and pieces of the
most wonderful, insightful trivia get buried inside internal reports
where people who could make use of the tidbits in understanding what
we've built never see them (who would have expected, for example,
that it would be possible to sustain average loads above 90% of full
over 10 minute periods on a T3 backbone circuit with less than a 0.1%
output drop rate using a router which had less than 8 milliseconds of
output buffer for the interface).  Of course there were always people,
who I didn't disagree with, who consistently asserted that many sorts of
useful data about the network they were a part of were no ones' business
but their own, but if you're kind of fond of learning what works and how,
and what doesn't work and why, it is less fun than it used to be.  Sometimes
it is hard to get rid of the bath water without dumping the baby with it.

I'd note one thing, however:

> growth and for planning.  Obviously several things in the past
> year have occured that have greatly shortened this time.  Amongst

In fact you can't know even this, since you haven't measured it you can
only guess.  It is always the case, when you're living on an exponential
growth curve, that this year's growth looks pretty humungous compared to
last year's, and next year's projections look even more amazing, but this
doesn't tell you whether the slope on a log-linear plot has actually turned
up or not.  And the growth of your own service over the past year in
particular is going to be a bit dangerous to extrapolate to anything
general from, since it has grown from a point near zero only a year ago
to really big now you've probably got severe divide-by-zero problems with
this data.

The data from years gone by at least have the merit that they were in
fact measured to be so.  If I were guessing I'd rather gamble on an
extrapolation of something I knew to be "true" last year, rather than
try to intuit how things might have changed this year, since intuition is
notoriously fallable.  I suspect this is why the 1 year doubling time
still persists.

Dennis Ferguson