IRE Laboratory tests: automatic systems chapter Robert N. Oddy Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Laboratory programs 165 Relevant Non-re[OCRerr]evant Matching value 2 3 m [OCRerr]"igure 9.3. Table for counting relevant and non- rclcvant documents at each matching value very different from modern online systems; although it resembles the early magnetic tape-based retrieval systems which processed queries in batches. If the researcher is concerned with aspects of the interaction between the user and the computer, then the additional complexity of the retrieval process will usually oblige him to adhere more closely to the structure of real life systems. An experiment to evaluate the effect of relevance feedback8'0, for instance, would require the program to select a retrieved set of documents, so that their indexing can be used together with relevance decisions to compute a modified query. No doubt, more subtle and efficient ways could be devised, but this may be self-defeating, in that one's confidence in the correctness of the programs may be endangered. Programs which model some aspect of the user's cognitive behaviour may have no formal specification: the details of the model are worked out in the process of writing, and trying the program. These programs will almost certainly be prototypes (see, for instance, Oddy33), inasmuch as they proceed as a real life system would, simply because you cannot design algorithms which efficiently capture the essence of a technique for test purposes until you know precisely what that technique is. Those who have been engaged in laboratory work in information retrieval over several years now have extensive suites of programs which can be applied in a wide variety of combinations to test collections. The well-known Smart system is described in Salton8. Programs are included for deriving the numerically coded form of a collection from natural language text, for clustering documents, for selection of clusters appropriate to a query (the whole collection may be selected), for searching the selected clusters, and for evaluation. All of these programs have a number of optional facilities. The program suites used at Cambridge University are described and discussed by Sparck Jones2 and Sparck Jones and Bates6. A major advantage of an automatic information retrieval laboratory, equipped with a flexible program suite and a number of conveniently formatted test collections, is that it is possible to try to accumulate convincing