Incompleteness
Since test collections can only be used to compare runs, don’t need complete judgments, just need unbiased judgments
- completeness is a way to guarantee fairness
- with appropriate controls, pooling provides a good approximation
- need to carefully evaluate effects of new pool building paradigms with respect to bias introduced