CRANV2 Test Environment chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 26 - comparison was made of 42 questions on aerodynamics and 42 questions on structures. Fig. 2.13 is a chart showing the distribution of the number of relevant documents throughout the 221 questions, from which it can be seen that the range is from six questions each having only one relevant document to one question which had forty relevant documents. Sample Precision Results At low precision ratios, the clerical work involved in obtaining correct figures was so great that in some cases it did not appear to be justified. This was due to the large number of non-relevant documents which would be retrieved and therefore had to be recorded. With index language I. la, results were obtained down to the single term level but with other index languages the decision was taken that, with the searches in the 1400-document collection, no attempt would be made to obtain precision figures below 5[OCRerr]0. This. however, introduced a variation between questions, since for a question having six starting terms, a precision figure lower than 5% might not be reached until the coordination level was down to two terms. However, with a ten starting term question, this figure might be reached by the coordination of four terms. In the presentation of the test results, note has been taken of this point and also the additional point regarding the number of questions capable of giving results, this being dependent on the number of starting terms which each question had. This can be best illustrated by referring to Fig. 2.14, which presents condensed results for 221 questions on the 1400-document collection with Index Language 1.2a. The column headed 'z' presents the figures for the number of questions that were potentially capable of giving results, and it can be seen that at a coordination level of 2, every question came'in this category. However, at a coordination level of 3, the total has dropped to 220, this indicating that there is one question which had only two starting terms. At a coordination level of 4, the total drops to 212, showing that there are eight questions with only three starting terms. As the coordination level rises, so the number of questions drops until, at a level of 15, it is seen that only three questions have thisnumber of starting terms. The column headed 'y' shows the number of questions which actually contributed figures for the calculation of the precision and fallout ratios - not. it should be noted, for the recall ratio which was always checked down to single term level. In Fig. 2.14, y is equal to z from a coordination level of 15 down to a coordination level of 7, and therefore the precision and fallout ratios can be based on complete data. However, at a coordination level of 6, only 161 questions were searched, and the precision and fallout ratios have been calculated on the basis of the non-relevant documents retrieved in these 161 searches. To indicate this, an asterisk