Research Seminars

Evaluation Music Information Retrieval (MIR) Systems: What's next?
J. Stephen Downie, Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign
Wed 19th March 2003

This presentation outlines a project to assist in the efforts being made to establish a TREC-like evaluation paradigm within the Music Information Retrieval (MIR) research community. The research tasks being proposed are based upon expert opinion garnered from members of the Information Retrieval (IR), Music Digital Library (MDL) and MIR communities with regard to the construction and implementation of scientifically valid evaluation frameworks. Major issues addressed in this presentation include comprehending the complex nature of music information; adequately capturing the complex nature of music queries; recognition of the "relevance" problem in the MIR context; and, the continued expansion of a comprehensive collection of music materials to be used in evaluation experiments.

Proposed solutions include the creation of data-rich query records that are both grounded in real-world requirements and neutral with respect to retrieval technique(s) being examined; adoption, and subsequent validation, of a "reasonable person" approach to "relevance" assessment; establishment of experimental protocols; and, the ongoing acquisition of music information (audio, symbolic and metadata) to enhance the development of a secure, yet accessible, research environment that allows researchers to remotely participate in the use of the large-scale testbed collection.

The seminar will be followed by a discussion about Music Grid and evaluation testbeds issues.