IRE Retrieval system tests 1958-1978 chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 250 Retrieval system tests 1958-1978 available online Systems, this is not to imply that they are perfect. We are, he suggests, `at much the same stage with mechanised information retrieval systems as was the automobile at the beginning of this century. All the essential ingredients are now there and working, but much effort is still required before we can have economical, reliable, and widely used models'. My view is that this is rather optimistic; Cleverdon's position can only be justified by a rather restricted engineering attitude to retrieval system research. Current systems can perhaps be better tooled, but they rely so heavily on the human user that the opportunities for a radical redesign of the rest of the system are limited. Changing the relative balance between the user and the rest might allow more interesting design possibilities; and to pursue these effectively we need more information than we have, which can only come from further theory development and testing. After all, while economic arguments may have recommended natural language to operational system managers, it is quite possible they would not have accepted these so readily without the intellectual confirmation provided by the results of experiments like Cranfield 2. It has taken quite a long time for the test results of the 1960s to filter through into practice, and we must therefore expect as slow responses to the experiments which have been carried out in the 1970s, particularly since Cleverdon's review, or which should be carried out in the 1980s. References I. CLEVERLON, C. W[OCRerr] Report on the Testing and Analysis ofan Investigation into the Comparative Efficiency of Indixing Systems, College of Aeronautics, Cranfield (1962) 2. CLEVERDON, C. W., MILLS, J. and KEEN, E. M. Factors Determining the Perlormance oflndexing Sj[OCRerr]stems, 2 Vols, College of Aeronautics, Cranfield (1966) 3. CLEVERDON, C. W. The Cranfield tests on index language devices, Aslib Proceedings 19, 173- 194(1967) 4. SALTON, G. A new comparison between conventional indexing (MEDLARS) and automatic text processing (SMART), Journal ofthe American Sociei[OCRerr]jor Inlormation Science 23, 75-84 (1972) 5. BOURNE, C. P. Evaluation of indexing systems. In: Annual Review of Information Science and Technology, Vol.1 (Ed. C. A. Cuadra), Interscience, New York (1966) 6. AITCHISON, J. and CLEVERnnN, C. W. Report on a Test of the Index of Metallurgical Literature of Western Resene Unilersity, College of Aeronautics, Cranfield (1963) 7. SCHULLER, J. A. Experience with indexing and retrieving by UDC and Uniterms, Journal of Documentation 12, 372-389 (1960) 8. CROS, R. C., GARDIN, J. C. and LEVY, F. L'Auhrnwtisation des Recherches Documentaires. Un Modele General: Le SYNTOL, Gauthier Villars, Paris (1964); 2nd edn with new preface (1968) 9. ALTMANN, B. A multiple testing of the natural language storage and retrieval ABC method: preliminary analysis of test results, American Documentation 18 33-45(1967) 10. BLAGDEN, J. B. How much noise in a role-free and link-free coordinate indexing system?, Journal [OCRerr]f Documentation 22, 203-209 (1966) 11. SHAW, T. N. and ROTHMAN, H. An experiment in indexing by word choosing, Journal of Documentation 24, 159-172 (1968) 12. LANCASTER, F. W. EI'aluation of the MEDLARS Demand Search Service, National Library of Medicine, Bethesda, Md (1968) 13. LANCASTER, F. W. MEDLARS: a report on the evaluation of its operating efficiency, American Documentation 20, 119-142 (1969) 14. CASE WESTERN RESERVE UNIVERSITY. An Inquiry into Testing oflnformation Retrieval Systems, 3 Vols, Comparative Systems Laboratory, Centre for Documentation and Communication Research, Case Western Reserve University (1968)