Table of Contents
Evaluation Infrastructure for Music IR: Lessons from TREC
What is TREC?
TREC Philosophy
TREC Impacts
TREC Impacts
TREC Tracks
TREC Tracks
TREC Tracks
Evaluation: How well does system meet information need?
Why do system evaluation?
Cranfield Tradition
Cranfield Tradition Assumptions
The Case Against Cranfield
Response to Criticism
Using Pooling to Create Large Test Collections
Data
Creating Relevance Judgments
PPT Slide
Test Collection Reliability
Inconsistency
Experiment:
Average Precision by Qrel
Effect of Different Judgments
Incompleteness
Incompleteness
Uniques Effect on Evaluation: Automatic Runs Only
Incompleteness
Cranfield Tradition
Cranfield Tradition
Implications for Music IR?
Possible Music IR Tasks
Task Implementation
Conclusion
|
Author: Ellen M. Voorhees
|