IRS13 Summary summary Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. search and retrieval methods follows in subsequent reports in this series. The analysis of information search procedures and the measurement of retrieval performance are tasks for which a clearly established method- ology does not as yet exist. For this reason, it becomes necessary to consider several different procedures, each designed to reveal a different aspect of retrieval evaluation: the total system viewpoint, the viewpoint of the user who insists on high precision, and that of the user who requires high recall. In each case, the aim of the studies included in the present report has been to reach conclusions which may be of practical help to the designer of automatic information systems. The analysis used is thus "insight oriented" rather than "proof-oriented" in the sense that a selective manual analysis of a few typical requests is used in order to gain an understanding of the general process. Formal statistical significance computations of the evaluation results are contained in section III of report ISR-12. All but three of the ten sections of this report have been prepared by E. M. Keen. Section I is devoted to a detailed examination of the test environment used, including a description of the document collections and search requests, of the manner of obtaining relevance decisions, of the variables entering into the evaluation process, and of the differences in the query and document make-up for the three collections in use. The evaluation parameters are examined in detail in section II. The viewpoint used in generating the performance measures is first described. This is followed by an introduction of some performance measures which are particularly useful for systems producing a ranked document output in de- creasing order of correlation between documents and search requests. Two types of evaluation measures are used including global measures where a xii