Response to Criticism
Goal in Cranfield tradition is to compare systems
- gives relative scores of evaluation measures, not absolute
- differences in relevance judgments matter only if relative measures based on those judgments change
Realism is a concern
- historically concern has been collection size
- for TREC, bigger concern is realism of topic statement