Test Collection Reliability
Recap
- test collections are abstractions of operational retrieval settings used to explore the relative merits of different retrieval strategies
- test collections are reliable if they predict the relative worth of different approaches
Two dimensions to explore
- inconsistency: differences in relevance judgments caused by using different assessors
- incompleteness: violation of assumption that all documents are judged for all test queries