Keynote/Boardwatch Results

Tue Jul 8 00:02:15 UTC 1997

I hate to say I told you so.  Actually, I don't, but I can be such a pain
in the ass anyway what difference does it make.

It would appear that everyone is pretty smugly satisfied by concensus that
the performance series we ran actually measures server performance and that

since all ISPs run weeny home servers, this was not "really" a test, flawed
methodology, etc.  I corresponded with Doug the Hump at Digex about this. 
I've liked this guy since I first met him largely because he's funny and
doesn't take himself too seriously.  He's got a yen for black helicopters
that still has me in stitches.

In any event, he didn't appear to be emotionally involved, but noted that
he did think their web server was the problem in their case and that it was
a real issue with the other backbones as well.  He said that if we had
measured one of their honking customer web servers that it would have all
been better.

Now I was clear with Doug, as I have been in this mailing list, that
servers DO have impact.  When you are attempting to measure end to end, you
certainly hope that the results cummulatively represent everything that has
an effect.  But I have been very clear that I didn't buy the "measuring
server" theory.  It will have an effect, but not near the effect you all
have apparently dreamed up amongst you with no data at all.

Anyway, Doug coughed the names of a couple of "honkin" customer sites.  One
on a UNIX machine running Apache.  One on an NT server.  As it so happens
they are the NIKE site and the FORBES site.  I agreed to run them all for a
few days and publish the results rather openly.  Here they are:

Www.digex.com

Metro Area	Abbr	Population	Mean	Std Dev	Data Pts.	Rating	
Omaha	OMA	640000	-	-	-	-	
Norfolk	ORF	1443000	0.917	0.787	477	-	
Milwaukee	MKE	1607000	2.641	5.545	427	-	
Cleveland	CLE	2860000	2.893	4.155	432	-	
Washington D.C	WAS	6727000	3.326	10.274	436	-	
Kansas City	MKC	1583000	3.334	3.881	440	-	
Detroit	DTT	5187000	3.424	11.593	470	-	
Atlanta	ATL	2960000	3.434	5.774	447	-	
Denver	DEN	1980000	3.497	2.725	433	-	
Tampa	TPA	2068000	3.541	9.753	440	-	
Minneapolis-St. Paul	MSP	2539000	3.697	7.263	420	-	
Pittsburgh	PIT	2395000	4.039	9.484	441	-	
Miami	MIA	3193000	4.336	11.227	409	-	
Chicago	CHI	8240000	4.433	7.977	432	-	
Philadelphia	PHL	5893000	5.687	36.621	446	-	
Columbus	CMH	1345000	6.582	10.275	352	-	
San Diego	SAN	2498000	7.124	6.563	111	-	
Houston	HOU	3731000	7.156	19.197	426	-	
Boston	BOS	5455000	8.147	68.522	342	-	
New York	NYC	19550000	8.911	48.549	436	-	
San Francisco	SFO	6253000	9.462	70.834	367	-	
Phoenix	PHX	2238000	12.226	26.802	447	-	
Seattle	SEA	2970000	14.989	74.818	201	-	
Dallas-Ft. Worth	DFW	4037000	16.24	36.844	421	-	
Los Angeles	LAX	14532000	33.879	78.169	572	-	
Salt Lake City	SLC	1072000	48.719	80.656	383	-	
Portland	PDX	1793000	65.23	88.641	446	-

			11.287	42.959	10654
www.nike.com

Metro Area	Abbr	Population	Mean	Std Dev	Data Pts.	Rating	
Omaha	OMA	640000	-	-	-	-	
Norfolk	ORF	1443000	1.359	6.704	385	-	
Washington D.C	WAS	6727000	2.838	6.486	426	-	
Cleveland	CLE	2860000	2.892	5.958	424	-	
Detroit	DTT	5187000	3.032	8.818	380	-	
Milwaukee	MKE	1607000	3.262	10.087	420	-	
Tampa	TPA	2068000	3.263	5.876	427	-	
Philadelphia	PHL	5893000	3.697	7.305	433	-	
Kansas City	MKC	1583000	3.82	7.477	429	-	
Los Angeles	LAX	14532000	3.979	7.142	441	-	
Denver	DEN	1980000	4.22	8.725	421	-	
Miami	MIA	3193000	4.338	13.774	398	-	
Pittsburgh	PIT	2395000	4.392	10.68	431	-	
Minneapolis-St. Paul	MSP	2539000	4.545	8.688	409	-	
Atlanta	ATL	2960000	4.577	16.279	438	-	
New York	NYC	19550000	5.163	13.025	427	-	
Boston	BOS	5455000	5.725	16.02	347	-	
Chicago	CHI	8240000	5.95	11.926	425	-	
San Francisco	SFO	6253000	7.254	21.74	353	-	
Houston	HOU	3731000	7.557	14.675	421	-	
Seattle	SEA	2970000	9.359	22.033	184	-	
Columbus	CMH	1345000	9.991	21.586	358	-	
Phoenix	PHX	2238000	14.089	25.896	432	-	
Dallas-Ft. Worth	DFW	4037000	17.677	44.361	407	-	
San Diego	SAN	2498000	19.029	10.833	16	-	
Salt Lake City	SLC	1072000	37.156	92.765	302	-	
Portland	PDX	1793000	79.404	166.148	437	-	

			9.868	44.374	9971
www.forbes.com

Metro Area	Abbr	Population	Mean	Std Dev	Data Pts.	Rating	
Omaha	OMA	640000	-	-	-	-	
Kansas City	MKC	1583000	0.0050	0.0	14	-	
Norfolk	ORF	1443000	2.165	11.616	380	-	
Miami	MIA	3193000	2.715	10.382	396	-	
Washington D.C	WAS	6727000	3.151	22.964	419	-	
Philadelphia	PHL	5893000	3.177	14.547	423	-	
Milwaukee	MKE	1607000	3.204	19.791	416	-	
Atlanta	ATL	2960000	3.364	17.068	434	-	
Minneapolis-St. Paul	MSP	2539000	3.847	10.974	405	-	
Cleveland	CLE	2860000	3.879	20.768	415	-	
Denver	DEN	1980000	3.974	22.303	417	-	
Tampa	TPA	2068000	4.001	19.877	421	-	
Pittsburgh	PIT	2395000	4.333	25.179	426	-	
Detroit	DTT	5187000	4.428	20.891	376	-	
New York	NYC	19550000	4.912	21.919	423	-	
Boston	BOS	5455000	5.347	30.106	340	-	
Seattle	SEA	2970000	6.324	12.105	179	-	
Chicago	CHI	8240000	6.378	26.431	421	-	
San Francisco	SFO	6253000	6.575	35.339	351	-	
Houston	HOU	3731000	7.49	20.885	417	-	
Columbus	CMH	1345000	7.726	19.924	351	-	
Dallas-Ft. Worth	DFW	4037000	13.068	22.705	404	-	
Phoenix	PHX	2238000	15.445	50.526	427	-	
Los Angeles	LAX	14532000	23.117	150.429	460	-	
Salt Lake City	SLC	1072000	47.801	227.832	297	-	
San Diego	SAN	2498000	85.378	153.109	14	-	
Portland	PDX	1793000	89.043	241.734	425	-

			11.53	79.068	9451

The bottom line is that there is some slight variation, but as I predicted,
not much.  And as it so happens, it was generally in the wrong direction. 
Nike was a little better on the mean and a little worse on the standard
deviation.  Forbes was a little worse (in fractions) from Digex on the
mean, and more so on the deviation.  Digex's original figures for the April
20- May 20 period were 9.162 seconds on the mean and 31.752 on the standard
deviation - slightly better than average.  Note that 30 days and five days
are apples and oranges if you comprendo fruit.  The means for the five day
period were roughly 9.9, 11.3, and 11.5 with the "weeny" Digex server in
the middle.

So yeah, servers do have an impact, but not nearly what you had hoped and
believed. I would say miniscule in a universe where our results ran from
1.5 to 26.8 seconds.  And while I'm not shy about "I told you so's" my real
reason for putting this out is that I have heard from several backbones
that are scrambling to upgrade and move their home page servers etc.  I
personally would get them up to what you think they ought to be anyway, but
if you go to extraordinary measures, you're probably going to be
disappointed in how little the numbers move - as I predicted.  It just
won't move the numbers much.  A little perhaps, and if you're not careful -
potentially the wrong direction.  Doug was pretty emphatic that these
customer servers were the "good" ones on the good part of the net, and the
home server was the weeny one.  Maybe Doug Mohney can jump in and remind me
which was which as far as NT and UNIX goes if anyone is interested.  I
would guess off hand that Forbes is taking a little more load than Nike,
but I may be reading messages from God in standard deviation cloud
formations.  There just isn't that much difference - certainly not in the
mean.

The bottom line is that if we are actually measuring server performance, we
should be able to measure three different servers, two avowed muscle boxes
and one avowed weeny one, all on the same network (connected differently
I'm told) and get at least as wide a variation as we saw between networks. 
We didn't by about a mile and a fortnight.  At LEAST it ought to be in the
predicted direction.  This clearly was not.  Good theory - but not so -
even in the lab.

Again, I think you guys should take a look at this stuff a little more open
mindedly and professionally.  It's certainly NOT to scientific laboratory
standards, but it is certainly interesting and I would claim very VALID
information.  Better information than you have previously had at your
disposal.  It's an attempt to look at the FOREST, not tree limb diameters,
leaf patterns and nutrient flows - all very interesting though those may be
I do grant you.  A great deal of this network has operated on theories that
once scaled up, nobody really knows if they work that way or not.  I can
tell you from personal experience that most of what I know is wrong, and I
find that out over, and over, and over again.  I might also mention that a
lot of what I'm told turns out to be wrong as well.  I'm only going to
SUGGEST that I may not be alone.

We'll continue to work on it.  I discussed the universal test page
suggestion with Gene Shklar this afternoon and we will make it so.  Again,
I don't think it will move any numbers around much, but certainly, as
Forest Gump says, ONE LESS THING....  And if you can make a case for a
different server ON YOUR OWN NETWORK, we will certainly entertain requests
to shoot at another machine.  Nominally July 15-August 15th though I'm not
signing up to those precise dates at this time.  

Regards

===================================================================
Jack Rickard                                    Boardwatch Magazine
Editor/Publisher                    8500 West Bowles Ave., Ste. 210
jack.rickard at boardwatch.com                     Littleton, CO 80123
www.boardwatch.com                             Voice: (303)973-6038
===================================================================