CRANV2 Supplementary tests and results chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 230 - Since this direct adjustment on the basis of generality does not equate performance, it is therefore necessary to consider whether N should be revised for the structures questions. It has already been established (on pages 71 to 76} that real N for the aerodynamics questions is not 14',)0, but in the region of 1027, at which figure the true generality number is 4.6. One might first hypothosise that the remaining 373 documents represent N for the structures questions; the corresponding generality number would be 16.3. To match this new figure, the adjusted precision ratio for the aerodynamics questions would now be 10.6%. which is higher than the figure {8.6%) for the structures questions. It therefore appears that in the collection of 1400 documents, there must be a subset which is common to both. Using the methods given on pages 71 to 76, N for the structures questions is shown to be probably at least 474, which gives a generality number of 12.8. Adjusted to this generality number the precision ratio for the aerodynamics questions is now 8.6%, the same as for the structures questions. The fallout ratios also now match; for the aerodynamics questions, where N = 1027, the fallout ratio is 9.2%; for the structures questions, where N = 474, the fallout ratio is 9.270%. The phrase "probably at least 474" was used because no account has been taken of the possibility that the performance figures will be affected by the comparative firmness of the terminology of aerodynamics and structures. The phrase, in fact, implied a belief that aerodynamics has the mushier or more imprecise language, and that for this reason, one would expect the set of structures questions to provide the better performance. However, the matter is complicated even if this latter point is ignored. At a coordination level of five, the structures questions have a performance of 16.1% recall and 18.1% precision. No exact matching figures can be obtained from Fig. 4.120T, but reference to 4.125P shows that, for the aerodynamic questions, at 16% recall, precision would be approximately 25%. Adjusted for generality.on the basis worked out earlier, this would increase tile precision ratio to 62%, which is far in advance of the figures for the structures questions. On the other hand at a single term level, it is found that the 42 structures questions have retrieved a total of 22,929 documents, which is an average of 538 documents for each question. This is a figure larger than the 474 documents earlier hypothosised as representing N. The above discussion is neither clear nor conclusive, and offers no explanation for the crossover in the performance figures of the two sets of questions {which is probably an 'otyerration caused by the relatively small number of results). Rather it serves to point up some of the difficulties which are involved in- attempting to compare performance in different subject areas by the coordination level cut-off, and emphasises the necessity for further research in this and related fields. Performance comparison by coordination levels In Chapter 4, all the tables of results and accompanying performance curves were based on the variation of coordination level. From these tables, sets of figures are extracted where the coordination level is held constant while the variable is the index language. Figs. 6.10T and 6.11T deal with the Single Term index languages at a coordination level of 3 and 6. Figs. 6.12T and 6.13T present the results at coordination levels of 2 and 4 for the Simple Concept index languages, while Figs. 6.14T and 6.15T present results at the same coordination levels for the Controlled Term index languages.