CRANV2 Methods for presentation of results chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 56 - important practical advantage, namely the comparative ease of calculation. To have used the method of the average of ratios would have increased the calculations forty-fold; work that has taken hundreds of hours would have taken hundreds of weeks. The really important matter in any test is to know which method is being used and to use it consistently in all situations. Method of totalling results Apart from deciding on whether to use the average of ratios or the average of numbers, we were faced with the additional problem which is involved in totalIing results of sets of searches where tim co- ordination level cut-off is employed; an idea of what is involved in this problem can be seen in Appendix 4A. There are given the performance results, in actual figures, for the 221 questions (subset 3), being tested on Language I. la (single terms, natural Ianguage and coordination), Exhaustivity 3, Search rule type A, Document Relevance 1 4 and searched on the 1400 document collection. The questions are arranged in numerical order, and for each question is given the total number of relevant documents in the whole collection, followed by the relevant and non-relevant documents actually retrieved at each coordination level. In the final column is given the sum of the total number of postings for the search terms; the total must, of course, equal the sum of the relevant and non-relevant documents at all coordination levels. The variations between the 221 questions that affect the problem of arriving at a single result of a single performance curve for the 221 questions can be seen in tabular form in Fig. 3.20T. Here the two characteristics of the 221 questions are listed, namely the numbers of terms initially selected from the search question and used as search terms (starting terms), and the number of retrieving terms, that is the maximum number of starting terms which, used in logical product coordination, may be put to the index and will still retrieve documents (whether relevant or non- relevant). The table shows how, for this particular.test, the starting terms ranged from 2 to 15, and the retrieving terms varied from 2 to 10. Within this 14 x 9 matrix the actual number of questions involved is recorded, so it can be seen, for example, that of the 35 questions having seven starting terms (column headed 7) only three of these questions could coordinate all seven terms and still retrieve some documents. The figures in the table refer.only to the particular index language in use, and a different index language such as index language 1.5 which includes synonyms, word endings and quasi-synonyms, would alter the distribution of the questions in relation to the retrieving terms, while any test involving a different basic index language (such as simple concepts as compared to single terms) would alter the starting term groups also.