INCOMING: IP Provider Metrics
mathis at zippy.psc.edu
Wed Mar 15 16:53:01 UTC 1995
Attached is my initial cut at an agenda and some introductory discussion for
the upcoming IP Provider Metrics BOF at Danvers. I plan to circulate this
to the main IETF list on Friday.
Even though I am going to work hard to prevent it, this is likely to become
quite political. My intent is to provide tools and methods (standard
procedures) which will enable the market to reward providers who are true to
their mission, what ever that may be.
Although my historical point of view has always been in the operations
community, I have clearly become "just a customer". Furthermore, there is a
large pool of "experts" who think that they can do the job better than you.
Therefor, it is crucial that current providers take an active role in this
effort, if nothing else to offset the "experts".
Comments are welcome.
Draft agenda of the "IP Provider Metrics" BOF
6:00 - 11:30 Wednesday, April 5, 1995
Consider a possible WG. (15 minutes at opening)
* - Is a new WG needed?
* - Relationship to BMWG/GISD/others
- Inventory of possible tasks
- Requirements for IP Provider Metrics
* - Define scope and priorities
* - Chair issues, Editor/secretary
* - Charter
(* introduce at the open, but postpone detailed discussion until the close)
General Requirements for IP provider metrics (15 min)
IP bulk data transfer performance (technical issues, 30 min)
IP routing stability and robustness (technical issues, 30 min)
Consider a possible WG...., part 2 (30 min)
---------------- Rough Draft Charter
IP Provider Metrics (IPPM)
The IPPM WG will develop a set of standard metrics that can be applied to the
the quality, performance and reliability of an IP datagram service. These
metrics will be designed such that they can be performed by the providers
themselves, customers, potential customers or independent testing groups.
The areas covered will include:
I) Path Performance
A) Bulk data transfer performance (ftp, etc)
B) Interactive performance (Telnet, X11, etc)
C) Real-time and multicast performance (delay statistics)
II) Routing stability and robustness
A) End to end availability
B) Time to recover (e.g. switch to backup paths)
C) Route stability (Spurious route transitions, etc)
III) others TBD
The IPPM WG will select or adapt existing tools and develop standard
procedures for performing and documenting the measurements.
---------------- Notes on the Charter
o Services peripheral to basic IP, such as NOC/NIS services, DNS, etc. are
expressly excluded because there is vehement anti-consensus among Internet
providers. Prior efforts to standardize these areas have failed because
they impinge on the different business models held by the providers. For
example some providers include full, high quality DNS service bundled with
their IP service. Others provide inexpensive DNS "starter" services, but
expect most customers to eventually run their own servers.
o It is also important that the language not be pejorative, to permit
providers to strategicly balance their price and performance. We want to
encourage all market positions including "Inexpensive, good enough service"
as well as "The best possible service".
o The milestones will be weighted to address I.A. (bulk performance) and
possibly parts of II first. The other goals will be addressed as an ongoing
effort. The scope will start very narrow, and will be extended only as we
have assured closure on the initial tasks.
o I have a prototype bulk transfer diagnostic that can safely and accurately
measure IP bulk performance. It is a combination of traceroute and Reno
TCP, and can provide traceroute like path information under exactly the
conditions induced by TCP. See the discussion below.
o The difficulty with routing stability metrics, is that the best ones require
collusion with the Internet providers themselves. This clashes with the
provider-to-provider trust model that is a necessary for healthy operation
of the Internet as a whole. BGP can easily be used to collect engineering,
business and other data on competitors. I believe that there exists a good
technological solution based on the global collection and archiving of
sanitized BGP logs. Clearly, there is a substantial piece of this
technology that belongs with the RA and other global routing agents.
However, there are also parts that belong to the community in the form of a
working group. There needs to be a discussion of the possible role for IPPM
in this process.
o The IETF is not "poised" to address "standard measurement procedures". Is
this a problem?
o The final draft of the charter should be completed after the WG is convened.
o IPPM should have an editor other than the chair.
Comments on the bulk performance tool:
I am working on a prototype diagnostic tool ("treno") designed to address bulk
performance issues. It is derived from Windowed ping (Matt Mathis, INET'94,
See ftp://ftp.psc.edu/pub/net_tools/WPing.ps), but designed to be safe for
typical network administrators at user sites.
It uses traceroute to emulate Reno TCP over any IP infrastructure. This
approach has a number of strong advantages:
- Its bandwidth consumption is bound by the same congestion mechanism as TCP.
(Although this is more than might be tolerated for ongoing monitoring).
- It measures the network under the same queue dynamics as inflicted by TCP.
- Differences in the end systems need not affect the results.
- Today it can identify the lowest performance provider in a long path if
there is a suitable unix box near each interconnect.
- Someday it will be able to diagnose (to a specific hop) any path across the
Treno is not yet completed. There are some open issues which can best be
addressed as part of an IETF WG, with open and public input from all interested
parties. In the end this will make it a stronger tool for supporting the
growth of the Internet.
- Certify that treno emulates Idealized Reno TCP. Understand and evaluate the
implications of differences between treno and Reno as it appears in the
field. (IPPM and end2end research community)
- Develop testing methodology for "data sheet" measurements: How to select
and specify endpoints, sampling schedule, etc for measuremets to be used on a
"data sheet" or service level language in a contract. (IPPM and the network
- Document the need and utility for all router vendors to support full rate
ICMP TTL exceeded messages. (IPPM and RREQ?)
More information about the NANOG