CRANV2 Test Environment chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. experimental test a set of environmental conditions has to be created, and some of those are inevitably, to a greater or lesser extent artificial. The software factors relate to the intellectual design of the storage and retrieval parts of an indexing system. The three main software factors are all the subject of management decisions in a given situation, and such decisions are always centred (although often unconsciously) on the twin parameters of exhaustivity and specificity (defined and discussed in Ref. 2). The operational factors are concerned with the routine operation of a system, i.e. all the processes required to make documents available to enquirers when the system has been set up. The factors in Fig. 2.1 are not intended to be an exhaustive list, but are given to illustrate the range of operations involved. Any basic evaluation of such factors is complicated by an infinite number of possible compromises between the least effort and the best quality, with both effort and quality being subjective notions notoriously difficult to measure. The hardware factors refer to the purely physical aspects of system operation that involve man-made entities. A brief and incomplete listing of five items is given. If one considers Cranfield II within this framework, it can be seen basically to have investigated the software factors, in the context of a laboratory situation in which the environmental factors and operational factors have been strietly controlled. Hardware factors have been ignored because, in this investigation, the measurements are being made on those software factors which are quite unaffected by changes in hardware. The operational factor of retrieval performance is the main measurement made, and details of how this is done are given in Chapter 3. In the artificial environment created for the test it was found that a limited set of changes could be investigated; these ineluded several sets of questions picked by different criteria, relevance judgements made in four different grades, collections of three different sizes and tests in two related but different subject fields. Software factors The software factors examined in the tests will be described and discussed first. A simplified table (Fig. 2.2) shows the variables that have been examined, listed under the three major factors of indexing, index languages and searching.