IRS13 Evaluation Parameters chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 11-29 simply interpreted by noting that a cut-off is established immediately when a relevant document is encountered in each output list. It very effectively reflects merit at the high recall end of the plot, since the lowest precision ratio for any individual request is computed when a recall of 1.0 is reached, unlike the 1'Pseudot' plot which continues making cut-offs until the last document in the collection is reached. This technique is quite adequate for making comparisons with in SMART, but a possible disadvantage in some cir- cumstances is that the curve is not typical of a real user environment since it produces too optimistic a result. Figure 16 b) compares a `Pseudo1T and a 11Quasi1 curve for the same set of averaged results. A modification to the technique is being tested, in which the cut-off having the highest precision at each vertical segment is still used, but the interpolation lines are altered to produce what is believed to represent the best possible curve that a user could achieve, assuming that almost optimum choices of cut-off are made. Figure 17 a) gives an example of this, and the reason for this type of extrapolation line which retains constant precision resides in the fact that user requirements would ask for the best possible precision above x recall. Whatever the value of 11x11 is, the best possible precision is always the next peak in the step curve, so a line of constant precision leading to that peak is thought to give the required result. A slight modification, yet to be made, is that sometimes, the next peak encountered above `IxTT recall is eclipsed by a higher peak at still greater recall (occurring, for example, when one relevant document is followed by another in the rank list). The line should thus be connected to the highest peak. This technique is known as the "Semi-Cranfield'1 method, and an average curve is presented in Figure 17 b), together with curves of the other two types. The comparison is slightly affected by a different tied rank procedure used for the "Semi-