CRANV2 Methods for presentation of results chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 32 - I00 9O 8O 70 60 50 4O 3O 2O I0 \ 0 I0 20 30 40 50 60 70 80 90 I00 0/o PRECISION x Relevance 2 o Relevance 2 and 3, FIGURE 3.1P PERFORMANCE CURVE OBTAINED WITH FACET INDEX IN W.R.U. TEST While the recall and precision ratios have been generally accepted as performance measures for information retrieval systems, they have also aroused some criticism. No serious attempt has been made to answer this criticism, partly because it was mostly trivial and never supported by experimental data, but mainly because an intention of Cranfield II was to investigate the performance measures which could or should be used. For this, sets of performance data were required and it was known that for every set of figures in Cranfield I, there would be hundreds of sets in Cranfield II, and it was obvious that the decisions regarding the measures to be used and the methods of presenting the test results would be of major importance. The programme of work which this aspect of the project has involved has been considerable, with many sets of results being calculated in a number of different ways. Based on this work, which has taken up a significant part of the effort during the last eighteen months of the project, the decision was finally reached that the most satisfactory method of calculating results involves three measures, namely Recall Ratio and Precision Ratio with, additionally, the new measure of Fallout Ratio. For the presentation of results on a plot, it is believed that, in the large