IRS13
Scientific Report No. IRS-13 Information Storage and Retrieval
Evaluation Parameters
chapter
E. M. Keen
Harvard University
Gerard Salton
Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government.
11-33
generality distributions.
The other suggested methods all use some technique of extrapolation,
so that all requests have full length precision recall curves that extend
from 0.0 to 1.0 recall. The second method involves extrapolation of the
beginning of all curves to 0.0 precision at 0.0 recall. Four examples using
different numbers of relevant and different rank positions are given in Fi[OCRerr][OCRerr]re
19. This method is justified mathematically, since if no documents are
retrieved (cases a) and c)) recall is 0, and precision is strictly zero, and
if the first document retrieved is non-relevant, recall is zero, and precision
zero ([OCRerr]l = 0). The disadvantage of this method is that the intermediate
values introduced by the extrapolation lines do not make much sense.
The third method uses extrapolation of all curves to 1.0 precision
at 0.0 recall, and is normally used by SMART together with the T1Quasi-Cranfield1'
recall level cut-off. Figure 20 reproduces the four previous examples processed
in the indicated manner. In documentary terms, when no documents are examined
(cases a) and c)) precision may in a sense be regarded as perfect, hence the
1.0 precision point is used. Cases b) and d) pose a problem for the precision
ratio, since retrieval of non-relevant documents only, normally indicates
zero precision, but the 1.0 precision ratio is used here for these cases also
for reasons of simplicity. As with the second method, the main disadvantage
is that intermediate values introduce[OCRerr] by the extrapolation lines have no
user-oriented meaning.
The fourth method is pr[OCRerr]posed in an attempt more correctly to reflect
precision in cases b) and d), where only non-relevant documents are retrieved.
Thus if no documents are retrieved at all a 1.0 precision and 0.0 recall
is used; but if [OCRerr]on-relevant documents only are retrieved ffrst, then 0.0
precision at 0.0 recall is used. Figure 21 gives the examples, but this hybrid
combination of methods 2 and 3 still provides poor meaning to a user.