ISR10 Evaluation of Document Retrieval Systems chapter Joseph John Rocchio Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 5-1 CHAPTER 5 EVALUATIOI[OCRerr] OF DOCITh[OCRerr][OCRerr] RETRIEVAL SYS[OCRerr]S 1. The General Problem An operationally effective automatic document retrieval system must satisfy the requirements of a diverse class of users. The functional model discussed in this thesis considers only a subset of these requirements, principally those related to the methodological aspects of the system operation. The ability to establish system requirements is directly related to the notion of performance evalua- tion, and in recent years the investigation of evaluation measures for document retrieval systems has rec[OCRerr]ed considerable attention (refer- ences 1 through 8). The purpose of this chapter is to consider in general the problems associated with the design of evaluation experiments and the collection of performance statistics, and to discuss in particular the implications of the system model on these problems. The general aim of a document retrieval system is to mechanize the deduction of the attribute 11relevance1,' which is a subjective relation between the state of a user's informatiQn need and the information content 6f the "docume[OCRerr]nts'[OCRerr] in some collection, as perceived' by the user. It is clear that since the' system operates The references cited are only representative of the literature on this topic.