Keynote/Boardwatch Internet Backbone Index

Michael Dillon michael at priori.net
Thu Jun 26 23:26:37 UTC 1997


[as posted to the NANOG mailing list]

>http://www.keynote.com/measures/backbones/backbones.html

Sigh...

>To measure backbone performance, Keynote downloaded 10,000 bytes from one web
>server on each
>backbone network and recorded how long this took, from the time the URL was
>requested until the bytes
>were downloaded from the server across the Internet to the measurement
>location. Downloads were
>attempted every 15 minutes, 24 hours a day, between April 20 and May 20, 1997,
>from each of the 27
>Keynote Perspective software measurement agents located in major metropolitan
>areas around the United
>States. Each complete measurement was multiplied by 5 to produce a result more
>representative of how
>long it would actually take a user to download a full web page with all its
>graphics. 

Sorry, you can't just multiply your measurements by 5 and expect to come up
with meaningful numbers. If you want to know the time needed to download a
full web page then you need to measure that specifically.

>Keynote decided to measure a backbone provider's own public web
>server on the assumption that the
>provider would locate its own server in the best-performing hosting
>location for that provider.

Whoa! Bad assumption. I know that in Priori's case, our web server will be
located in the WORST performing location because the performance of our web
server is not critical to our network's performance. Our office will be
connected to our local POP via a T1 whereas a colo customer would be in the
same room as a backbone router. I wouldn't be surprised to see similar
scenarios elsewhere, especially among the large providers who have built
out special web farm colo space with the best connectivity. As always, the
shoemaker's children are most likely to be in bare feet.

This is an interesting way to measure end-to-end performance but these two
flaws make this particular set of data completely meaningless.

If you want to measure the service that a backbone provides to its
customers then you really need to use a customer server, not the backbone
operator's own public server which may be located in-house at their
offices. And you need to use real-world example pages that are replicated
to each of the servers. And, quite frankly, I don't think that automated
data collection really can provide a clear picture of the relative
performance of the different backbones. There are too many different things
that can happen on the network and it doesn't make sense to average them
all together. The methodology doesn't mention whether or not outliers in
the data were discarded and what steps they took to identify those
outliers. The study as it stands does not accurately reflect the quality of
the user's experience and that seems to be the original intent of this
study.

Keynote touts itself as the "premier supplier of software and services for
measuring and managing the responsiveness of web-based applications" and
yet they do not even appear to be aware of the tools and methodologies
available from the IPMA project http://www.merit.edu/ipma or from CAIDA
http://www.caida.org or some of the other end-to-end performance
measurement info from sites listed in the "Tools" section at
http://www.nlanr.net/INFO/






********************************************************
Michael Dillon                    voice: +1-415-482-2840
Senior Systems Architect            fax: +1-415-482-2844
PRIORI NETWORKS, INC.              http://www.priori.net

"The People You Know.  The People You Trust."
********************************************************





More information about the NANOG mailing list