IRS13
Scientific Report No. IRS-13 Information Storage and Retrieval
Evaluation Parameters
chapter
E. M. Keen
Harvard University
Gerard Salton
Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government.
11-50
viewpoint result in a precision-recall graph, since an equation to adjust
precision for generality is given in (2], namely,
Adjusted Precision Ratio = RlxG
(R x G) + F (1000 - G)
1 1
where R = Recall ratio at a given cut-off point
1
F Fallout ratio at a given cut-off point
1
G = Generality number (1000 x total relevant) to which
collection size
it is desired to alter the results.
Thus, in Fig. 28(c), the ADI recall and fallout ratios are recorded as R1
and F for a series of cut-off points and G is set to 2316, in order to
1
adjust the generality of ADI to fit the generality of Cran-l. The adjusted
precision versus recall curve is given in Fig. 28(c). It should be noted
that the precision for ADI does not now represent a user-oriented evaluation,
but has been artificially adjusted to give a system oriented evaluation. A
series of tables appears in (2] in which the fallout values for ranges of
recall and precision values have been computed, for a range of generality
numbers, primarily to permit quick calculation of adjusted precision ratios.
Some comparisons involving changes in generality are given in
section I and Appendix A, and further comparisons using the Cran-l and larger
Cran-2 collections will require performance measures of this type. It should
be emphasized, however, that the ordinary precision-recall curve still gives
a vali[OCRerr] and useful user-oriented result, and it is in experimental test com-
parisons only that the two viewpoints for evaluation (Fig. 2) give different
and complementary results. The normalized evaluation measures appear to
reflect a system-oriented result since the equations both contain "N", the
total number of documents in the collection. For example, the normalized