IP Performance Metrics WG (ippm)
Monday, 27 March 2000, at 19:30 - 22:00
=======================================

The meeting was chaired by the working group chairs, Will Leland
and Matt Zekauskas, and was reported by Paul Love.

AGENDA
1. Status
2. Loss patterns
3. IPDV experience
4. Periodic streams
5. Bulk transport capacity
6. Futures

IETF home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM home page: http://www.advanced.org/IPPM/


1. Status, Will Leland (Telcordia)

IPPM opened with a status report.  Since the meeting in Oslo there
were minor changes to loss patterns and IPDV, a new draft on delay for
periodic streams, and the expiration of the Treno bulk transport
capacity metric.


2. Loss Patterns, Rayadurgam Ravikanth (Nokia)

Ravikanth noted that minor changes have been made since Oslo
(a draft was released in October addressing the Oslo comments),
and that no further discussion had occurred on the mailing list.
If there are no further changes, one more editorial pass will
occur just after IETF, and then the authors wanted it to be submitted
for consideration as an Experimental RFC.

A question from the floor asked if the draft had been implemented,
with any experience using the metric.  Other than some internal use by
the authors, there's been no published work using it.  There was
a presentation of the ideas by example at the LA IETF -- see
those minutes.

A question from the floor asked what it means for a loss to be
"noticeable".  The mathematical definition seemed clear, but the
motivation was not.  The explanation was that a stretch of time might
be a problem if repeated small numbers of losses, even though no small
group would be a problem.  The number of losses, the number of
intervening successful packets are parameters and not specified in the
draft.  It depends on the application what would be considered
noticeable.  For details, see Rajeev's thesis referenced in the
draft.  Another audience member asked if there could be some
guidance on how to use the metric, perhaps pulling the gist of the
thesis description into the draft.  Ravikanth thought this would
be a good idea.

With no other comments, the authors will add another example, make
final editorial changes, and will ship a draft in about two weeks
following IETF.  We will then "last call" the document as
experimental.


3. IPDV, Matt Zekauskas (Advanced Network & Services), presenting

See slides.

Matt first gave a summary of the changes to the document.  Then he
presented Phil Chimento's slides on the IPDV draft.

Phil had experimental results to show using the metric applied to
Surveyor data.   There was solid consensus that the I-D has made
the right decision to condition on received packets, that is, reporting
losses but not including them in the IPDV values.

Phil noted that the histograms were first cuts, with the histogram
buckets evenly distributed over the entire min-max range.  Because
there are extended tails, Matt Mathis suggested that the bucket size
should be allowed to increase as one moves away from the center
of the peak.

Another observation made on the sample data was that the min and
max IPDV value often occurred close together.  This is consistent
with spikes developing in queues along the path.

There was extensive discussion of whether IPDV's differential approach
was better than deviation from an expected (or first) value.
Dr. Guven Mercankosk was concerned about glitches in display of
real-time media streams.  He felt that the variations reported by the
differential approach do not reflect the buffer size required.  (The
buffers would be too small.)  In the views of the chairs, the rest of
the audience felt that measurements represent a point in time and do
not guarantee future variation.  One example raised was a shift in
route during a session.  In this case, an arbitrarily large buffer
might be required to guarantee perfect display.  A histogram of the
differential variation would show one outlier; a histogram of the
"expected" approach would be bimodal.  The requirement depends on the
application.  There was consensus that the differential approach was a
sufficient initial approach.  Experience with real applications using
this metric will be required.  The chairs encouraged Dr. Mercankosk to
present his case to the mailing list so that the author (and the
working group in general) could respond.

A suggestion was made to provide some information on usefulness
of IPDV, and any obvious limitations in the document.

There was also discussion of the relationship between the delay
variation measured by the two methods (differential versus expected).
It was agreed that while they are related, you cannot directly
derive one from the other without having the complete time series
of variation measurements (and a known starting point).

At several points during the meeting, Dr. Mercankosk suggested that
interpreting the metrics will sometimes require knowledge of the
background traffic.  The example given was configuring an application
based on measurements taken at the start of a session.  Someone
mentioned that if you need a hard guarantee, then you need QoS.  The
relevance of background traffic to defining metrics was not well
understood (since if more aspects of the network should be captured
then additional metrics should be defined), and Dr. Mercankosk was
encouraged to present the concern to the mailing list.


4. Periodic Streams, Chuck Powers (Motorola), presenting

Chuck Powers presented the new I-D on "periodic streams" for the authors,
Vilho Raisanen and Glenn Grotefeld.  The chairs suggested that the key
questions were if a periodic streams metric was needed, and if the
proposed metric was a good approach.  The meeting easily agreed that
periodic traffic is increasingly important and can exhibit delay
behavior not seen by Poisson samples.  Voice over IP, streaming audio
and streaming video have regular traffic.

This document is parallel to the one-way delay document, but for
periodic flows.  The mapping to the Framework needs to be improved.
The mailing list notion of a "sample of samples" was discussed.
The "sample of samples" refers to multiple streams
that are sent over time to better characterize what an arbitrary
periodic stream might have encountered.  A Poisson schedule of
the periodic streams was agreed to be a good approach.

Dr. Mercankosk thought that the sample of samples would indeed
provide more information about the underlying network, but again
called for better characterization of background traffic.

In response to a question by Will Leland, Matt Mathis and Vern Paxson
noted that periodic streams see distinct loss behavior in addition to
delay behavior.  For example, periodic streams may synchronize with
slot times for cable modem access.  Another example is what is known
as a "clock slip" in telephony, which causes a point-to-point link to
drop bits periodically.  This led to a discussion of periodic versions
of all the existing metrics.  There was some consensus that periodic
streams needed to be considered for all IPPM metrics, but no
volunteers to address this larger question.


5. Bulk Transport Capacity, Will Leland & Matt Zekauskas

See slides.

The Co-Chairs then described the progress, or lack thereof,
with a bulk transport capacity metric.  The BTC framework
was updated in October, but the Treno draft has now expired.

Mark Allman now has concrete plans to produce a draft of
'cap' as a BTC metric by the end of the summer.  'cap' is
a user-level TCP emulation tool, originally built to easily
experiment with changing TCP features.  Mark is planning
to perform experiments that tests the relationship of the
values cap produces to the implementation of TCP in FreeBSD.

Matt Mathis described what blocked Treno progress: Treno does not
tolerate any packet reordering, unlike most TCPs.  His concern was
that a metric that allowed any packet reordering might create a
situation where vendors designed for that reordering.  ISPs
that use such equipment might pass TCP throughput tests.  However,
reordering is cumulative.  A path that included a concatenation
of ISPs, each of which reordered a few packets, would 
produce a stream of reordered packets that TCP would not tolerate.

There is a high demand for measuring TCP throughput and
many poorly designed ad hoc approaches are used today.
One is web browser download time, which conflates DNS lookup
time, server load and network throughput.  It is also at
the mercy of the end-host TCP stacks.  Another approach
is to just bless one stack, perhaps the Windows 2000 implementation.
Both approaches are vulnerable to future software changes.

There was wide consensus that a well-founded metric, even
if imperfect, is needed quickly.  The chairs asked if anyone
had an alternative approach they would advocate, but
none was suggested.


6. WG Future, Will Leland & Matt Zekauskas

See slides.

The meeting closed with a wide-ranging discussion of future directions
for the group.  To date, the focus has been on defining a small set of
unambiguous useful metrics that focus on transport, not applications.

Will noted that the original reason behind IPPM was to develop
accurate measurements for service level agreements and specifications.
Others noted that these have a "net to net" rather than "host to host"
focus.

A related concern was the perceived bias of IPPM metrics toward active
measurements.  The applicability of the existing delay and loss
metrics to passive observation of actual traffic has not been
resolved.

Another issue raised was metrics for the emerging distinct classes of
service in IP.  Ruediger Geib mentioned the idea of back-to-back
active measurements where each has a different class of service, in
order to quantify the difference made by that class of service between
two points.

Other potential directions raised during the discussion were
comparison of metrics, metrics for network availability, MIBs for
these metrics, and some way of defining meaningful "weather maps",
rather than the ad-hoc maps used today.

The consensus was that the working group needs to continue to actively
develop further metrics.  The working group provides a forum to debate
new metrics; without it ad-hoc metrics evolve without careful
scrutiny.  Will closed the meeting by noting that there were several
good directions mentioned today.  We will be looking for volunteers to
define them; if there are no volunteers the WG would go dormant.