CRANV2 Test Environment chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 24 - indexing records. The choice of the subset of .200 documents extracted from the 1400 was governed by:- 1. Use of a set of aerodynamic questions and documents. 2. Choice of the largest set of questions that could be tested on a subset of 200 documents. 3. Choice of questions restricted to those not haviug any relevant documents in the range of Numbers 1001 1299 (because of the different weighting method used at that stage of the indexing). The third rule restricted the choice of questions quite considerably, and the second rule was modified by not allowing 'simiiar' questions - mainly two or more questions having an overlapping set of identical relevant documents asked by the same questioner. The 42 questions finally used had 198 relevant documents in the subset of 200 documents. None of the base documents to the 42 questions were included. The second subset chosen was one of 350 documents (subset 2), and this subset included the 200 documents of subset 1. Subset 2 also consisted entirely of aerodynamic documents, with an additional 35 questions having all their relevant documents in the subset. These, to- gether with the 42 questions for subset 1, resulted in the 77-question subset (subset 7) which was used for the tests on the controlled vocabulary. In presenting the test results, the majority of results are based on these smaller subsets of documents and questions. The first tests were made with 221 questions on the 1400 collection, and these tests were repeated on smaller documents and question subsets in order to validate the use of such subsets. It will be shown in the next chapter how the difference in performance can be adequateiy accounted for, and the use of smaller subsets does not, we believe, impare either the validity or, to any appreciable extent, the accuracy of the results and findings. The use of these subsets enabled the various environmental factors involved to be investigated. For example, the effect of the change in collection size from 1400 to 200 documents with a fixed set of questions was investigated. Comparison was also possible between the 350 and 200 collections. In the case of the questions, different subsets were made up and results obtained when environmental factors such as those listed in Fig. 2.11 were being investigated. The four grades of document relevance are included in the main test results (Chapter 4) and the effect of the other factors that are listed is considered in Chapter 6. An attempt was made to compare the two distinct subject fields that existed in the 1400 document collection. Many of the question sets contained both aerodynamics and structures questions, but the direct