CRANV1P1 Documents and Questions chapter Cyril Cleverdon Jack Mills Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 39 - Co -ordination Level Cited Documents Additional Documents Total Recall Total Recall Recalled Ratio Recalled Ratio 99 94.3% 128 92.8% 80 76 2% 101 73.2% 59 56.2% 75 54.3% 40 38.1% 46 33.3% 17 16.2% 25 18.1% 9 8.6% 10 7.2% 2 1.9% 3 2.2% Total Relevant 105 138 TABLE 3.12 COMPARISON OF RECALL PERFORMANCE OF RELEVANT 1CITED' AND ADDITIONAL DOCUMENTS IN RELATION TO 25 QUESTIONS. All questions had seven s*,arting terms; the table shows the effect on recall of increasing the search requirements from any one term to all seven terms. Using the same set as considered in the previous paragraphs, the 25 questions which had some additional relevant documents were used, cOmparing 105 cited with 138 additional documents. Here again the difference between the two groups is not significant, (see Table 3.12). For instance, at a coordination level of 2, the recall ratios are 76% and 73% for cited and additional documents; at a coordination level of 5, the figures are 16% and 18%. These results {which are, of course, only a small sample of what will be presented in a later report ) should have revealed any un- natural question-document bias, whether conspicuous in the title or not, had any bias been present at all. We are confident that there is no measurable unnatural match between the questions and the documents themselves. Questions obtained from a real life situation and tested on an existing coliection might give different results in some way, but until such a test is done, and a comparison is made of different test methodologies, it is not possible to state in what ways, and by how much, the present test method falls short of the ideal in this respect.