IRE The Cranfield tests chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 13 The Cranfield tests Karen Sparck Jones As the broad survey of Chapter 12 suggests, Cleverdon's Cranfield 1 and 2 projects have played an extremely important part in the development of retrieval system experiment. The purpose of this chapter is to discuss their contribution in more detail. I shall consider the objectives, form and results of the two projects in turn, and the reactions to them, concluding with a discussion of their influence on retrieval testing as a whole. Cranfield 1 and 2 were major tests from every point of view[OCRerr]in their scope, their conduct, and their impact[OCRerr]and as such must be examined. I shall not, however, attempt an exhaustive review of all the Cranfield-related literature, but rather focus on the two projects in their contexts, referring as appropriate only to the more important comments on them. I shall not attempt either to discuss all of Cleverdon's own papers, but will refer only to those bearing on the Cranfield 1 and 2 tests as experiments. In Cleverdon's own work Cranfield 1 was preceded by the collaboration with Thorne, and was paralleled by the English Electric test and the joint Cranfield-Western Reserve University experiment, referred to in the previous chapter as Cranfield l[OCRerr]. Cranfield 2 was followed by the DOAE and Precision Engineering tests, and more recently by the NASA study. Together these tests constitute a major contribution to information retrieval experiment, showing those features which may be said to mark the Cranfield tradition in the methods and concerns of retrieval experiment, but also reflecting the major change in retrieval systems brought about by automation. In this chapter, the account of Cranfield 1 is followed by a discussion of related work and of the reactions to the test, and by an assessment of these reactions; Cranfield 2 is then treated in the same way. This approach has been adopted to make the position of the tests in their historical contexts clear, but it must be emphasized that at Cranfield itself the second test immediately followed the first, and that the intellectual continuity between the two was strong: Cranfield 2 was very much addressed to questions which arose during Cranfield 1 as well as to dealing with early criticisms of Cranfield 1. It should also be noted that there have been changes in standard terminology since the early Cranfield work: in particular in relation to performance measurement relevance is now generally called precision, and what was originally referred to as efficiency would now, given the way the 256