IRS13 Evaluation Parameters chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 11-6 3. Measures for Ranking Systems The provision of a ranked output, in which documents are ordered according to the magnitude of their correlation coefficient with the search request, m[OCRerr]s it possible to use evaluation measures of many types, since a direct evaluation of the rank positions occupied by the relevant documents may be made, or a series of cut-offs may be applied according to many dif- ferent criteria. The requirement of a unique rank position for every document in the collection in SMART does require some procedures for dealing with tied ranks. Specifically, there always exist some documents that exhibit a zero correlation with the search request; these documents are given a rank position according to a random procedure in order that complete performance merit can be measured. Although relevant documents rarely take up such rank positions in the more usual processing runs, the use of titles alone does result in larger numbers of such cases. Also, tied rank8 may occur with possitively correlated documents particularly when the overlap correlation is in use; this, however, occurs quite rarely with the better cosine correla- tion runs. The performance measures used by SMART are now briefly described, and some additional suggested measures noted. The primary purpose of measurement in &MART has been that of internal comparisons (Purpose 1, Figure 2), and the viewpoint that of user satisfaction(Viewpoint 2, Figure 2). A) Single N[OCRerr]ber Measures Sets of measures known as Rank Recall and Log Precision, and Nor[OCRerr][OCRerr]alized Recall and Normalized Precision are in use, and have been des- cribed [l,[OCRerr],5,6]. These measures are cut-off independent in that the rank positions of all the relevant documentB to a request are compared with the