CRANV2 Main test results chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 140 - Section 6 Document Relevance The four grades of document relevance were first tested on the 35 questions (subset I), and results are given for languages I.l.a and 1.6.a. Figs, 4.600T - 4.605T give the tables of results, and Figs. 4.606P - 4.607P the plots of performance curves. The change in the different relevance decisions causes a change in the generality number, which is shown in each table, and therefore recall/fallout plots are presented. Although the set of questions used was typical, the test results of the two higher grades of relevance (1-2 and l) suffer from the fact that not all questions had any relevant documents of these grades. 27 questions had relevant documents when grade I-2 was tested, but only 11 questions when grade I was tested. All 35 questions, however, were kept in the set and the non-relevant documents retrieved by these questions were still counted. It can be noted that the change in document relevance from low to high merely transfers Some of the relevant documents from the relevant retrieved to the non-relevant retrieved category, while the total documents retrieved always stays constant. Because of the small total of documents having relevance 1 documents, a further test was made on a set of fifty questions, for which the criterion of selection was that each question must have a relevance 1 document. These were tested on the 1400 document collection with index language I.l.a and the results are shown in Figs. 4.610T - 4.613T, with a reca]A/fallout plot as Figure 4.614P. LIST OF FIGURES Relevance 4. 600T 1-3 4. 601T 1-3 4. 602T 1-2 4.603T I-2 4. 604T 1 4.605T i 4. 606P i -4 1-3 1-2 and I 4. 607P 1-4 I-3 I-2 and I 4.610T 1-4 4.611T 1-3 4,612T 1-2 4.613T 1 4.614P 1-4 1-3 I-2 I Index No. of Question Document Language Questions Subset Come.ion I.l.a 35 1 1400 1.6.a 35 1 1400 I.l.a 35 I 1400 1.6.a 35 I 1400 I.l.a 35 1 1400 1.6.a 35 1 1400 I.l.a 35 1 1400 1.6.a 35 1 1400 l.l.a 50 9 1400 I.l.a 50 9 1400 I.l.a 50 0 1400 I.l.a 50 9 1400 l.l.a 50 9 1400 Plot s Plot 4.110T 4. 600T 4.602T 4. 604T Plot 4,114T 4.601T 4. 603T 4. 605T Plot 4. 610T 4,611T 4.612T 4.613T