CRANV2 Methods for presentation of results chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 71 - With all the methods that have been discussed and consistently show d [OCRerr]v , ........... illustrated, each , e Ind_,, [OCRerr],,[OCRerr]uuge i. la to nave a seemingly superior performance to Index Language 1.6a. Whatever weakness there might have been in any of the methods, in no case were the results sufficiently distorted to mask this change in performance. In this situation, it again seemed most sensible to adopt a method which was relatively simple to apply, and Method 1A, using starting term coordination levels was therefore selected. After this decision had been taken and the main sets of results had been prepared, a simpler method of obtaining a ranked output was found, and the majority of the results have been recalculated by the document output cutoff method. However, the decision to present the main results by Method 1 was not reversed, so the results obtained by this alternative Method 6 are presented separately in Chapter 5. In view of the decision to use the starting term coordination level method, it is necessary to mention one further point. Using this method means that average results obtained at high coordination levels are based on an increasingly smaller number of questions in the set due to two reasons - firstly the variation in the number of starting terms, resulting in questions with a small number of starting terms never being capable of contributing results when the coordination level exceeds the number of starting terms. This has already been discussed in Chapter 2, where it was stated that this information was given in each table of test results in column z (see Fig. 2.15). The second reason was the variation in the number of terms that actually retrieve any documents, since the higher coordination levels in some questions demand a match that is too strong for any documents to be retrieved. Data on this point is presented in Column x which gives the total number of questions which actually retrieved any documents. As can be seen in Fig. 2.15, although z decreases at the higher coordination levels, x is smaller than z at all coordination levels of 4 or more. This was the normal experience, since there were usually some questions where the demand for a coordination of four terms would not retrieve a single document. The generality number 1,000(a+c) N To return to the matter of the generality number, it is now possible to consider this in more detail. It is known that, in situations where the generality numbers are different, varying performance figures will be obtained, even though the actual operational performance may be similar. In experimental tests such situations exist when the average numbers of documents relevant to the questions differ in two cases of identical file size or, vice versa, where the file sizes are the same but the numbers of relevant documents are different. A third situation is where both the numbers of the relevant documents and the file sizes are different.