ISR10 Evaluation of Document Retrieval Systems chapter Joseph John Rocchio Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 5-20 retrieval operation may be expressed as follows: a retrieval operation with respect to a request q is expected to produce an ordering on the reference collection D, such that every member of the set D is ranked R above all members of the complement of D with respect to D(DR) [OCRerr]ote that in this formulation no emphasis is placed on any relative order among the members of the set D of relevant documents. R While such an ordering might in theory seem desirable, the determination of an unordered set D is difficult enough by itself, so that imposition of an additional ordering criterion[OCRerr]may be impractical. A parti[OCRerr]l order within D may, however, have some significance and, in fact, has been R employed in some of the ASLIB-Cranfield experiments to specify degrees of relevance. These in turn lead to the definition of different subsets DR, but not to the specification of retrieval order with respect to relevance order. Given the previously stated definition of the objective of a retrieval operation, two functions. of the ordering induced on D may be defined which are related to the recall and relevance (precision) of Cleverdon. Consider an ordering induced on D by M such that a one-to- one mapping exists from D to the dense set of integers from 1 to increasing rank order in the set of integers then reflects decreasing connection between the request image and document image. In this case, define: -[OCRerr] for 1< i < n0 r*(i) = 1 for n < i < [OCRerr] L -