IRE The Cranfield tests chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 282 The Cranfield tests alternative relevance assessments25, and Lesk and Salton's study26, showed that comparative performance was not materially affected by different relevance assessments. Second, the test was not concerned with specific, established indexing languages like the UDC, or language types like facetted classifications, so the results could not be regarded as threatening by language proprietors and advocates. At the same time, insofar as the devices studied might be regarded as associated with types of language, it is likely that Cranfield 1 had softened up the potential opposition. Third, other work, notably by the Smart Project, suggested that the results were not eccentric, but could be paralleled for other data, with alternative test designs. Fourth, an increasing interest in the use of natural rather than artificial, controlled indexing languages, both for intellectual reasons among research workers and practical ones among the new computer-based system operators, meshed in with the Cranfield 2 results. An additional reason was perhaps the realization that anyone wishing to subvert the Cranfield findings, either in terms of a laboratory test or operational system investigation, would have to do a great deal of work. For Cleverdon personally, Cranfield 2 was followed on the one hand by controversy about the recall/precision relationship27, and on the other by involvment in tests like the DOAE28 and Precision Engineering29 studies of 1970, and the NASA experiment30 of 1977, described in Chapter 12. These tests clearly show their descent from Cranfield 1 and 2, in being concerned with comparing index languages in relation to language control or indexing descriptions in relation to description exhaustivity; but they also represent studies of indexes and indexing in new environments: the NASA test for instance involved online searching. The underlying continuity of Cleverdon's work is also apparent in his interest in costs, which repeatedly figures in his discussions of system evaluation. In the research community there were two direct responses to Cranfield 2. One was the application of Cranfield 2 principles and practices in other tests, usually selectively, or with modifica- tions, but perceptibly: the Inspec tests31 and Keen's ISILT project32 are good examples of this trend. Other projects, like that of Sparck Jones33, utilized Cranfield 2 performance representation methods, for example. The second response was to exploit the Cranfield 2 collection, once it had been made available in machine-readable form. A great many Smart Project experiments were carried out with the Cranfield data34, and the collection was also used by, for instance, Svenonius35 in the US and by Sparck Jones and Bates36 and van Rijsbergen and Croff37 in the UK. It is more difficult to assess the indirect influence of Cranfield 2 than of Cranfield 1. Substantively, Cranfield 2 suggested that retrieval systems may not work very well and are difficult to upgrade significantly. But this was being suggested by other tests in the later 1960s as well. Methodologically, it is very probable that Cranfield 2 encouraged care in the analysis of systems and the conduct of experiments, but by the later 1960s the same point was being made by other projects too. Cranfield 2 did not, moreover, lead directly into work on those topics which were regarded as most important in the 1 970s, namely the effects of searching on performance and the design of mechanized systems. But that its indirect influence was great can be inferred from the fact that so many research workers in the late 1960s and [OCRerr]970s referred to Cranfield.