Editor's note: These minutes have not been edited. Minutes of the Benchmarking Methodology Workgroup Report by Kevin Dubray Approximately 45 people attended this meeting. 1. Agenda. The proposed agenda for this BMWG was accepted as follows: - Administration - Discussion of Proposed Cell/Call benchmarking - Discussion of LAN Switch Draft 2. Administration. Kevin Dubray introduced himself as co-chair of the BMWG, replacing Jim McQuaid. Jim was thanked for his service. Dubray also reminded attendees of the meeting time and place of the IP Provider Metrics (IPPM) half of the BMWG. It was announced that the "Benchmarking Methodology for Network Interconnect Devices" draft was approved as an informational RFC 1944. It was reported that a discrepancy had already been found in RFC 1944 with regards to a range of addresses. Scott Bradner informed the group that the error was reported to the RFC Editor. As of the meeting, Scott had not heard back. A question was raised on what to do in the interim with anomalies found in BMWG documents. It was offered that a list of "Outstanding Issues" would be offered periodically through the BMWG mailing list and stored via the archive. Dubray volunteered to manage this. Dubray mentioned that the BMWG charter and goals were updated to better reflect the other BMWG effort, IPPM, chaired by Guy Almes. The charter can be accessed via the URL: http://www.ietf.cnri.reston.va.us/html.charters/bmwg-charter.html 3. Workplan - Proposed Cell/Call Benchmarking. An assignment from the last BMWG meeting, Dubray presented his action item for a proposed Cell/Call Benchmarking workplan that was previously distributed on the mailing list. Given the cell focused benchmarking activities of other organizations (ATM Forum, ITU-T, etc.), the scope of the proposed workplan focused on the definition of the terminology and methodology of benchmarking cell- forwarding/connection-oriented devices in PACKET-BASED internetworks. A vote was made on whether to proceed with the workplan. There was consensus that the group move forward with this effort. The workplan suggested that the effort be split into two distinct parts: 1. Draft a document identifying or defining the relevant terminology and metrics in characterizing the performance of these devices. 2. Draft a document defining the methodology of collecting the metrics produced in the Terminology document. Item 2 would only occur after consensus was reach on Item 1. A volunteer was solicited to head the Metric and Terminology effort. Robert Craig gratiously volunteered. Robert is to make an initial draft available on the mailing list. 4. LAN Switch Benchmarking Draft. Bob Mandeville was on hand to discuss the latest revision of the LAN Switch Benchmarking Draft. This draft has evolved significantly since its last version. In the latest version, Bob modified the draft to: - Focus on the identification of related metrics and terminology, much like RFC 1242. - Expand the draft's scope beyond Ethernet switches. There was a discussion on how various media will be supported, given the inherent differences between, for instance, deterministic media and contention-based media. Bob's response was to have a master document that would attempt to address issues in a most generic fashion. When specific topics warranted a more focused discussion, separate memos addressing these issues could be issued. An issue was brought up that most of the suggested metrics' stimuli were monolithic in style; further, the stimuli were offered outside the context of any application. It was cited that application-specific tests introduced many variables that made characterization difficult (e.g. How does running a file transfer on a 8086 PC vs Pentium-based PC using different protocol stacks, affect a switch's benchmark?) Another example, however, was offered citing the need to distinguish how a switch handles that file transfer in light of, say, video traffic. It was agreed that reliance on one specific application or another wasn't necessarily desirable. However, that is not to say that the use of generalized, application-like stimuli (e.g., the bursty nature of data transfers vs. more "constant" bit rate of video) isn't undesirable. Further, it may be desirable to see how the application of one stimulli impacts the other across the tested system. On the draft's discussion of bursts, Bob was cautioned that it was one thing to say, "Traffic is bursty,"; characterizing traffic as a "burst of one," is saying something else. The multidimensional traffic item was brought up. Several examples were offered (e.g., broadcast, multicast, Full Duplex Ethernet) on why it was difficult to characterize the behavior of a system with the use of multidimensional traffic. The consensus appeared to be that this is a useful concept if and only if the impact of the stimuli on the tested system could clearly be ascertained and communicated. Bob asked whether jitter should be a addressed in the document. A comment was made to the effect that if you consider jitter, you should consider both the VBR and CBR cases, but only if you can clearly describe the observed behavior. Bob queried the group as to whether the document should attempt to define a switch. The group generally agreed that there was more marketecture than technology these days in the taxonomy of a switch. Further, if there was enough distinct qualities between a generic switch, a Layer 2 switch or a Layer 3 frame/packet switch, subsequent work could address those features specifically. It was offered that regardless of the "type" of LAN switch, most of the concepts offered in the current draft could be applied. In general, there was a positive air regarding the draft. Bob was going to put the latest draft on the mailing list for further discussion. The draft should be discussed enough on the mailing list so that by the next meeting, the draft can be forwarded for consideration as a RFC. 5. Assignments & Goals: The next round of goals slated for the BMWG by the next meeting are: 1. Present for discussion a preliminary draft on the Benchmarking Terminology for Cell Forwarding/Connection-Oriented Internetworking Devices. (Robert Craig) 2. Prepare the LAN Switch Benchmarking Terminology Draft to the point where its ready to forward for consideration as a RFC by next meeting. (Bob Mandeville) The meeting concluded with Dubray requesting that people monitor and participate in the discussions on the BMWG mailing list so that expeditious progress can be made in the above areas.