CRANV1P1 Test Design chapter Cyril Cleverdon Jack Mills Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 17 So far the discussion on the test design has been entirely concerned with the methods to be used in obtaining a set of test documents and questions, and establish- ing the relationship between the documents and the questions. All such activity was an essential preliminary to the investigation itself, the general background of which . was considered in the previous chapter. To summarize this briefly, we started from the belief that all index languages are amalgams of different kinds of devices. Such devices fall into the two groups of those which are intended to improve the recall ratio and those which are intended to improve the precision ratio. In other words, there are some devices which will always enlarge the class and thereby retrieve more documents, with the probable result that more relevant documents will be re- trieved. On the other hand, the precision devices will always act in the reverse manner by narrowing the class, thereby retrieving fewer documents, with the probable result that some relevant documents will be eliminated. The purpose of the test was to investigate the effect which each of these devices, alone or in any possible com- bination, would have on recall and precision. To enable this to be done, it was essential that it should be possible to hold every- thing constant except the one variable being investigated. The organization of the file, with its completed matrix of document/question relevance assessments, was the first step towards this. The next stage was to determine and fix, once and for all, the concept-indexing of the documents and the relationships of the concepts. By concept-indexing ie meant the decision as to which concept and groups of concepts are significant from the viewpoint of retrieval. Such concept Indexing can only be in . the terminology of the document. As soon as there is any 'translation' of the document terminology to any kind of formalized language, then one of the index language devices must have been brought into use. Therefore the decision was to concept-index, at a high level of exhaustivity, the documents in the collection so that they might be translated into any type of index language which it was desired to test. Details as to how this was done are given in chapter 4. The original proposal to the National Science Foundation contained the following statement. "At this stage it should be possible to decide which technique appeared to have the most satisfactory characteristic for adaptation to automatic indexing. Dr. J. O'Connor has explained the techniques which can be used to investigate methods of automatic indexing without actually using computers, (ref. 19) . Our approach would be partly to investigate new techniques, but might as usefully be concerned with testing methods proposed by others and measuring the performance of such methods against the results from human indexing. " The possibility and the hope that the test collection could be used by other groups and provide direct comparison with the Cranfie:ld results was partly responsible for the decisions concerning the indexing technique and also the searching method. This permitted starting from. the absolute basic point of matching any actual word in the question with any term used in the concept-indexing and then to introduce all the devices by stages. It is agreeable to be able to record that it will be possible to compare the results of the Cranfield tests with two experiments using computers. In England, at the Cambridge Language Research Unit, the complete set of Cranfield indexing is being processed on the Atlas computer, and it will be possible to measure and compare the effectiveness of the 'clumping' process which Dr. Needham has been investigatingtref.29) In the United States, at the Harvard Computation Laboratory, a sub-set of the index- ing has been processed by a number of the options of the SMART programme which Professor G. Salton has designed. There is particular interest in this work, in that,