The Case Against Cranfield
Relevance judgments
- topical similarity is not utility
- static set of judgments cannot reflect user’s changing information need
- vary too much to be the basis of evaluation
Results on test collections are not representative of operational retrieval systems