IRS13 Evaluation Parameters chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 11-48 a similar number of relevant documents in both collections, (4.9 ADI, 4.7 Cran-l), the differing collection sizes (82 ADI, 200 Cran-l) show that the concentration of relevant items favors the ADI collection. This may be observed by imagining a user, who examines every document in both collections in order to be certain of gaining 1.0 recall, and who will finally end up with a precision ratio of .0592 in ADI, and .0236 in Cran-l (at this cut-off point, the precision ratio becomes the generality number itself). Thus higher pre- cision ratios are expected with higher generality numbers, at all cut-off points in the curve, unless some other factor such as subject language or request and relevance decisions causes some strong effect over the low and middle re- call regions of the curve. Fig. 28(a) includes a comparison of this type using the ADI (Docu- mentation) and Cran-l (Aerodynamics) collections, where the expected merit is found above 0.7 recall; below that point, the ADI collection falls below Cran-l. The reasons for this result are not important to the present dis- cussion relating to measurements. From the user viewpoint, the comparison in Fig. 28(a) accurately reflects merit, but from a system viewpoint, the change in generality number makes the ADI collection more hospitable to good retrieval than Cran-l; a measure is thus needed to take this into account. As was suggested in part 2, the value of "d" (Fig. 1, total non-relevant items not examined) is needed for system comparisons, and Fig. 1 also defines the fallout ratio as used in the Cranfield Project (2]. Fig. 28(b) gives a fallout recall graph of the ADI and Cran-l results, which shows that Cran-l is now correctly superior over the whole performance range, except at 1.0 recall where both curves meet. It is also possible to represent this system-