CRANV2 Conclusions chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 252 - CHAPTER 8 CONCLUSIONS The first step in testing a theory (qua theory) is to examine it to see what deductions can be made from it - to set up post- ulates which may be tested either experimentally or by observ- ations of the 'real-life' situation. That is to say. the first step in testing a theory is to state the practical consequences of it. If the deduced practical consequences (operational definitions) are proved to be unsustained, the theory is discre- dited. No theory can ever be proved to be true; it is held for so long as no better theory can be found. L.T. Wilkins: Social Deviance Page 36 Although the results presented in this volume inevitably represent only a condensation of the tens of thousands of individual results which have been obtained, it is hoped that they are in sufficient detail for anyone interested to make their own interpretation. It might, therefore, be argued that much of this final chapter is redundant, and that it would be better to leave readers to reach their own conclusions. However, the following comments are offered as a personal contribution, with the hope -and expectation - that others will feel free to deduce and argue. The results have been presented in three main ways. Firstly, there are the details of the search results for the various index languages, recall and precision devices and search rules as obtained with the conventional coordin- ation level cut-off. Secondly, some of these results have been regrouped to illustrate various aspects of the test and thirdly, many of the test results have been re-calculated by the document output cut-off method based on simulated ranking. While the opinions presented in this chapter may be illustrated by referring to a particular set of figures, they are not usually based on a single result. Within the definition as given in Chapter 2 of Volume 1, every set of figures supports the original hypothesis of an inverse relationship between recall and precision. It is immaterial which variable is changed to give a new system; it may be the coordination level (e.g. Fig. 4.100T), the exhaustivity of indexing (e.g. Fig.4,912P),the recall devices (e.g. Fig. 6/0T),the precision devices (e. g. Fig. 6. t 7]-), the search programmes (e. g. Fig. 4. 850T), or the relevance decisions (e.g. Fig. 6.3P); it has been impossible to find any exception to what can be claimed as a basic rule. Quite the most astonishing and seemingly inexplicable conclusion that arises from the project is that the single term index languages are superior to any other type. This is mainly evidenced by the results based on the normalised recall ratios of Fig. 5.15T, but also, although less obviously, by the comparison of different systems using the conventional coordination level cut-off (see Fig. 6.2P). This conclusion is so controversial and so unexpected that it is bound to throw considerable doubt on the methods which have been used to obtain these results, and our own first reaction was to doubt the evidence. A complete recheck has failed to reveal any discrepancies, and unless one is prepared to say that the whole test conception is so much at fault that the results are completely distorted, then there is no other course except to attempt to explain the results which seem to offend against every canon on which we were trained as librarians.