Keynote/Boardwatch Internet Backbone Index A better test!!!

Jack Rickard jack.rickard at boardwatch.com
Mon Jun 30 16:14:41 UTC 1997



I'll try.  Let's talk about them a bit one at a time.

 
> Fine, I will point out a few specific problems with your methodology:
> 
> 1.  Although a variety of backbones is used, the study does not say
> which ones.  Also, even though the study does point out the assymetrical
> routing of a web transavtion (hot-potato), it doesn't point out that
> the traffic being measured is a brief web request (which is dumped to
> the web server's backbone ASAP) answered by a long response (10KB in
> this case, dumped to the querier's backbone).
> 


We're trying not to care here.   I think the "point of view" entailed is
more along the lines of:  IF I had MY web server on a particular backbone,
either by running a dedicated access link to my office from that backbone,
or by taking advantage of co-location, or by actually having that backbone
host and manage my web site on their server, what would my web site look
like to the Internet universe - the people that are out there downloading
pages.  I think there is an ongoing attempt by netheads to reduce this to
flows between trunk routers.  We're looking at a backbone a bit more
broadly.  It is the cummulative total of your network, your connectivity to
other networks, your peering, your people, your coffee pots.  We don't want
to be drawn down into it much beyond that.  If I have a web site, using any
of various means, connected or otherwise hosted on YOUR backbone, what will
the performance look like to MY audience.  Will there be
differences/advantages to being on one backbone or another, with regards to
the perceived performance or rapidy of the pages appearing.  The answer to
the latter appears to be yes.


> 2.  The test used measures the responsiveness of a company's web servers,
> which is not necessarily reflective of the response their customers get.
> This test specifically measures traffic going "outbound", but suggests 
> that this information is useful in determining a carrier for "inbound"
> traffic.  This could be misleading; a web farm will have a lot more 
> outbound traffic than inbound, and a dial-up only provider will have more
> inbound traffic than out.
> 

Yes, it is being read as an indicator from the users perspective in the
inverse direction.  I suspect there IS a relationship, but we're a little
unclear on how closely coupled that is.  Certainly we're aware that traffic
from a customer to a web site takes a completely different path than the
almost always larger flow from the web site to the customer - this is
largely the reason we dismiss PING and TRACEROUTE as almost totally useless
for our purposes.  Do the relative performances we are seeing for backbones
translate into relative performances for, say, dialup customers connected
to that network?  I think we'll find yes here too and I don't really agree
with the implication that there is no relationship here.  But you are quite
correct, that is NOT what we intended to measure.


> 3.  The results show an average taken over 24 hours, ignoring the
important
> "time of day" factor.  Compuserve, for example, may be dog slow
> for three hours per day, and greatly over-provisioned for the rest of the
> day.  In other words, if over a three-hour period samples took 15
seconds,
> but only took 2 seconds for the rest of the day, it would score better
> than a provider with a consistent 6-7 second rate.  
> 

This is actually a very good point and has caused no end of discussion
internally.  In fact, I think it is one of the more interesting things we
have accomplished here.  The press release of course posted average
download times.  We also graph standard deviations and from my perspective,
this is an infinitely more interesting item, though harder to explain.  The
Internet is a very different place at 2:00 AM than at 2:00 PM, and in fact
from moment to moment.  There is a slightly larger standard deviation
waveform that implies consistency across various metropolitan areas.  The
standard deviation measurements show us how consistently a network can
deliver the same performance from one moment to the next, from one time of
day to the next, and to a lesser degree, from one city to the next.  We are
going so far as to somewhat disingeniously lable this PERFORMANCE UNDER
LOAD in an attempt to communicate this without a course in statistics.



> 4.  The transfer rate of 10KB x 5 is not the same as the transfer rate
> of a 50KB file.  If one backbone is significantly "burstier" than 
> another, this could dramatically affect throughput.  For instance, a
> 10KB file might easily go through a bursty or bouncy backbone in just
> a few seconds, while larger files require greater consistency.
> 

True enough.  But we applied the same methodology to all backbones equally.
 My sense of what makes up web pages is that they are rarely a solid 50 KB
of anything, but a series of files with a text file, a smallish logo file,
a couple of graphics files, etc.  We think readers can relate to the 50 KB
total, and scaled it so.  But we think most are made up of a series of
files between 5 and 20 KB.  I would think "bursty" networks would be a plus
in such an environement.  What I think I'm hearing you say is that latency
counts.  I agree.  I think it should, but again, we want to look at a
communicable "whole page" concept for the test download.

Characterizing just what a "typical" web page is is of course a rather
loose business.  We're pretty open to suggestions of specifically what the
download would be like, but it does need to be reasonably of a least common
denominator. 

> 5.  Some companies have more popular web pages than others.  Few major
> providers hang servers directly off their backbone (whatever that might
> mean in this context), but rather have a lobe or two attaching their
farm.
> Just because a provider's web farm is saturated or busy or slow, does not
> indicate that the rest of their backbone is.

The web server is operated by and under control of the network, as all
other aspects are.  The concensus here seems to be that we are primarily
measuring web server performance and not the network itself.  I'm trying to
let you all gel around this as the main objection.  In five days I think I
can show you that it is an interesting theory, just not so.

> 
> Looking back, some of these errors reflect a misinterpretation of mine
> that the results were intended for consumers looking for a connection, as
> opposed to a place to host their web server.  Most of these points still
> apply, however.  And people will use the study results this way; if I can
> make that mistake, J. Random LUser could, too.

This assumes there is no correlation.  I'm not sure I'm willing to concede
that at this point.  But in general, I can say that few of the national
backbones emphasize their dialup offering to the home user.  Most are
pushing web hosting and dedicated access connections to businesses.  If J
Random LUser wants to shop for a NATIONAL BACKBONE to get his connection
to, it does rather invert the concept.  You are absolutely correct that it
would be better, if that were the objective, to pick a dialup pop on each
backbone, and measure to ALL the backbone web sites from a single point and
then compare those results.  Easy enough, but not really what we're after
here.  In fact, there is no dialup component to this at all, and I'm
guessing dialup users would be more interested in accessibility and busy
signals anyway.

We do think it is quite interesting that the average delivery speed over
the backbone is little, if any, beyond the bandwidth capabilities of the
new 56K modems.  It might appear that an extraordinary amount of resource
is going toward upping the bandwidth to the home, when the perceptual
"speediness" of the world wide web may not improve at all until the
backbone performance increases.

> 
> Also, the links from the page to the graphs were diappointing.  What 
> units are measured in the "Best Value" chart?  What does the price of a
> T1 have to do with web hosting?  

We have rather lumped dedicated access in there wholesale.  Web hosting
pricing has no standards, it is a chaotic mix with some pricing by hits,
some by storage, some by the amount of data passing, some by flat rate. 
The "basic element" of pricing seems to be the monthly recurring cost of a
1.544 Mbps T-1.  

The actual formulation is of course in the article.  Basically we make the
loose assertion that performance is the most important criteria when
shopping for a connection.  But it is rarely the case that "price is no
object."  We somewhat gratuitously came up with the 2/3 ratio.  We sought a
formula where 2/3 of the buying decision was made on performance, and 1/3
on price.

What we are referring to as TOTAL PERFORMANCE is a simple multiplication of
the average download times by the standard deviation.  The average of THIS
total performance measure across the 29 backbones was 492.  The average
monthly price of a T-1 was $2045.  So we took each networks total perf and
multiplied by 8 in an attempt to approximate a value of 4000.  We then add
the monthly price of a T-1 to that to derive a VALUE figure.  Savvis
Communications had the lowest (best) VALUE number.  They scored well on
performance, but their $1700 monthly price was considerably lower than
CompuServe, AT&T WorldNet, UUNET, or even GridNet.  We named them the best
value on that basis.

> 
> All of the previous notwithstanding, I would be interested in a better
> version of this study.  Even more interesting would be to track how
> providers do over the course of several studies--who responds well to
> backbone congestion?

We are caught between two missions here.  In the first place, we are very
receptive to suggested improvements to the study.  I'm looking at
proving/disproving how much of the "web server" component we are really
measuring.  And we've taken quite seriously the concept of providing each
backbone provider a totally identical "test page" for example.  I don't
think the latter will seriously move any numbers, but as Forest Gump says
"just one less thing...".  At the same time, I think it will be intensely
interesting to see how the numbers move from one issue of the directory to
the next.  We will have backbones seeking to score better, but at the same
time, they will be under pressure from new Internauts and more web pages
added to the Internet daily.  Traffic growth appears to be continuing
unabated.  And the more consistently we measure from issue to issue, the
more true those relative numbers will be comparing one measurement period
to the next.

Jack Rickard

===================================================================
Jack Rickard                                    Boardwatch Magazine
Editor/Publisher                    8500 West Bowles Ave., Ste. 210
jack.rickard at boardwatch.com                     Littleton, CO 80123
www.boardwatch.com                             Voice: (303)973-6038
===================================================================




More information about the NANOG mailing list