Goal: Validate TREC Collections As Laboratory Tools
Why TREC collections?
- most commonly used test collections today
- significantly larger (> 500,000 docs)
- more relevant docs (some questions > 500 rels)
- used to compare more diverse retrieval systems
Validate how?
- show changes in relevance judgments don’t change comparative evaluation
- focus is on whether individual researcher can rely on the collections