IRE The methodology of information retrieval experiment chapter Stephen E. Robertson Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Bibliographic notes 31 Indeed, the best `further reading' list must surely be the rest of this book. fowever, some specific matters discussed in the text must be given sources. The reports of the Cranfield 2 and Medlars experiments are: (`I EVERnoN, C. W., MILLS, J. and KEEN, E. M. Factors Determining the Performance of Indexing .%`[OCRerr]tems (2 Vols), Aslib Cranfield Research Project, College of Aeronautics, Cranfield (1966) I AN( `ASTER, F. W. Evaluation of the MEDLARS Demand Search Service, National Library of Medicine, Bethesda, Md (1968) Another research report of about the same vintage which discusses the iiiethodological problems in some detail is: (`AN':. WESTERN RESERVE UNIVERSITY. An Inquiry into the Testing of Information Retrieval Systems ([OCRerr] Vols), Comparative Systems Laboratory, Centre for Documentation and Communication Research, Case Western Reserve University (1968) .[OCRerr]() me of the general issues are discussed in textbooks, e.g.: I AN('ASTER, F. W. Information Retrieval Systems: Characteristics, Testing, and Evaluation, Wiley, New York (1968) %AI TON, G. (Ed.) The SMART Retrieval System: Experiments in Automatic Document Processing, I[OCRerr]rentice-Ha1l, Englewood Cliffs, N. J. (1971) `I'he various experimental and theoretical investigations of relevance are [OCRerr]urveyed in: [OCRerr]ARACEVIC, T. Relevance: a review of and a framework for the thinking on the notion in ii'l.ormation science, Journalofthe American Society for Information Science 26, 321-343 (1975) `F he other experiments are reported as follows: Oddy's experlment: (I )I)v, R. N. Reference Retrieval Based on User Induced Dynamic Clustering, Ph.D. Thesis. (`omputing Laboratory, University of Newcastle upon Tyne (1975) `II (I)V, R. N. Information retrieval through man-machine dialogue, Journal 9fDocumentation 33, I 14(1977) I I'trter's experiment: IIARTER, S. P. A Prohahilistic Approach to Automatic Ken'[OCRerr]'ord Indexing. Ph.D. Thesis, Universitv [OCRerr])f Chicago (1974) ARTER, S. P. A probabilistic approach to automatic keyword indexing, Journal ofthe Amcrican .[OCRerr])ciety!o'r Information Science 26, 197-206 and 280-289 (1975) I'()rtable test collections: [OCRerr]I'ARCK JONES, K. and VAN RIJSBERGEN, C. 3. Information retrieval test collections, Journal of l)ocumentation 32, 59-75 (1976) There is no useful reference on the general nature of the statistical problems[OCRerr]they tend to emerge as asides or specific issues in the Various i'csearch reports such as those noted above. Another such report which considers some of these problems is: kEEN, E. M. and DIGGER, 3. A. Report of an Information Science Index L[OCRerr]nguages Test (2 Vols). (`ollege of Librarianship, Wales, Aberystwyth (1972) `l'he problems derivtng from the fact that there are two populations are [OCRerr]liscussed in somewhat theoretical terms ln: ROBERTSON, S. E. A Theoretical Model of the Retrieval Characteristir's of. Information Retrieval Systems, Ph.D. Thesis, University of London (1976)