ISR10 Evaluation of Document Retrieval Systems chapter Joseph John Rocchio Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. ½ with imperfect representations of both the' needs of its users and the information content of documents, and since the notion of relevance is /10 likely to be quite variable over any realistic user population, any performance measure must represent a statistical estimate of the probability distribution of a correct assessment of relevance. The subjective nature of relevance implies that any realistic system1s evaluation will require a large amount of data. Since the collection of such data is costly, and since the notion of evaluation is a critical element of any design process, the designer of a retrieval system must rely on analytical tools, local performance measures, and intuition to select, the most likely set of[OCRerr]functions to `satisfy his objectives. In this connection, computer based simulation systems such as SMART can be of significant benefit since they allow large sm6unts of data to be generated and analyzed. One of the major -. objectives of the functional model td be considered here (based on the SMART simulation system) is to `iiaximize[OCRerr]the utility of the data generating capabilities of the sy'stem by allowing evaluation of the individual functional elements `as well' as of the overall performance characteristics. 5-2 2. [OCRerr]valAxation Measures and the Collection of Statistics A. The Idealized Exper[OCRerr]iment `Most of the evaluation measures proposed for document retrieval systems are based on the follQwing idealized characterization