IRS13 Evaluation Parameters chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 11-33 generality distributions. The other suggested methods all use some technique of extrapolation, so that all requests have full length precision recall curves that extend from 0.0 to 1.0 recall. The second method involves extrapolation of the beginning of all curves to 0.0 precision at 0.0 recall. Four examples using different numbers of relevant and different rank positions are given in Fi[OCRerr][OCRerr]re 19. This method is justified mathematically, since if no documents are retrieved (cases a) and c)) recall is 0, and precision is strictly zero, and if the first document retrieved is non-relevant, recall is zero, and precision zero ([OCRerr]l = 0). The disadvantage of this method is that the intermediate values introduced by the extrapolation lines do not make much sense. The third method uses extrapolation of all curves to 1.0 precision at 0.0 recall, and is normally used by SMART together with the T1Quasi-Cranfield1' recall level cut-off. Figure 20 reproduces the four previous examples processed in the indicated manner. In documentary terms, when no documents are examined (cases a) and c)) precision may in a sense be regarded as perfect, hence the 1.0 precision point is used. Cases b) and d) pose a problem for the precision ratio, since retrieval of non-relevant documents only, normally indicates zero precision, but the 1.0 precision ratio is used here for these cases also for reasons of simplicity. As with the second method, the main disadvantage is that intermediate values introduce[OCRerr] by the extrapolation lines have no user-oriented meaning. The fourth method is pr[OCRerr]posed in an attempt more correctly to reflect precision in cases b) and d), where only non-relevant documents are retrieved. Thus if no documents are retrieved at all a 1.0 precision and 0.0 recall is used; but if [OCRerr]on-relevant documents only are retrieved ffrst, then 0.0 precision at 0.0 recall is used. Figure 21 gives the examples, but this hybrid combination of methods 2 and 3 still provides poor meaning to a user.