IRE The Cranfield tests chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. References 283 13.5 Conclusion What, then, is the Cranfield legacy? First, and most specifically, it has proved very difficult to undermine the major result of Cleverdon's work, namely that indexing languages, including natural language, tend to perform niuch the same: the gross substantive result of the research remains true. Second, methodologically, Cranfield 2, whatever its particular defects, clearly indicated what experimental standards ought to be sought. Third, our whole view of information retrieval systems and how we should study them has been manifestly influenced, almost entirely for the good, by Cranfield. But none of this means that retrieval system testing is wholly well organized. Cranfield 1 and 2 raised questions about the replication of tests results, and more significantly, their extrapolation to a large scale, which have not been answered. Moreover, as Rees said of Cranfield 2, the work does not provide system design instructions. The major gap in the Cranfield work was indeed the absence of any models which could underpin design recommendations: there was certainly some `ur-theorie' underlying the Cranfield tests; but it's a long way from ur-theory to theory proper, and we have so far only taken a few steps along the road. References I. ThORNE, R. G. The efficiency of subject catalogues and the cost of inforrnation searches, Journal of Documentation 11, 130-148(1955) 2. CLEVERDON, C. W[OCRerr] Report on the First Stage of an Investigation into the Comparative Efficiency of Indexing Systems, College of Aeronautics, Cranfield (1960) 3. CLEVERDON, C. W[OCRerr] Report on the Testing and Analysis of an Investigation into the Comparative Efficiency of Indexing Systems, College of Aeronautics, Cranfield (1962) 4[OCRerr] AITCHISON, j. and CLEVERDON, c[OCRerr] w. Report on a Test of the Index of Metallurgical Literature of Western Reserve University, College of Aeronautics, Cranfield (1963) 5. LANCASTER, F. W[OCRerr] and MILLS, J. Testing indexes and index language devices, American Documentation 15, 4-13 (1964) 6. SWANSON, D. R. The evidence underlying the Cranfield results, Library Quarterly 35, 1-20 (1965); this paper includes a comprehensive bibliography on earlier Cranfield-related literature 7. MOTE, L. J. B. Review of CLEvERDON, C. W. The Cranfield 1 1962 Report, Journal of Documentation 19, 8081(1963) 8. RICHMOND, P. A. A review of the Cranfield Project, American Documentation 14, 307-311 (1963) 9. SHARP, J. Review of the Cranfield-WRU Test Literature, Journal of Documentation 20, 170- 174 (1964) 10. TAUBE, M. A note on the pseudo-mathematics of relevance, American Documentation 16, 69- 72(1965) 11. HERNER, S., LANCASTER, F. W. and JOHANNINGSMEIER, W. F. A case study in the application of Cranfield system evaluation techniques, Journal of Chemical Documentation 5, 92-95 (1965) 12. KYLE, B. R. F. Information retrieval and subject indexing: Cranfield and after, Journal of Documentation 20, 5569 (1964) 13. CASE WESTERN RESERVE UNIVERSITY. An Inquiry into Testing of Information Retrieval Systems, 3 Vols, Comparative Systems Laboratory, Centre for Documentation and Communication Research, Case Western Reserve University (1968) 14. REES, A. M. Review of a Report of the Aslib-Cranfield Test of the Index of Metallurgical Literature of Western Reserve University, Centre for Documentation and Communication Research, Western Reserve University (1963)