IRE Laboratory tests: automatic systems chapter Robert N. Oddy Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 9 Laboratory tests: automatic systems Robert N. Oddy 9.1 Laboratory experiment in information retrieval A laboratory is a sheltered place where one constrains the enthusiastic abandon of the real world in order to attempt to lay the blame for one apparently capricious phenomenon upon another. The extraordinary diffi- culties experienced in all areas of science when designing experiments testify to the complexity of the interactions between natural phenomena. Experi- ments are concerned with relationships between events or states as reflected by the measurements that we choose, and are able, to make. It is invariably necessary to control, or at least to keep tabs on, factors which are not the immediate concern of the experiment, but are suspected to have some influence on those that are. These comments apply whether an experiment is conducted in laboratory conditions or not. What, then, are the characteristics of a laboratory experiment which distinguish it from a real life experiment? I do not think that there is a clear distinction. However, one might point to a tendency for the experimenter to initiate events at his convenience in the laboratory, whe?eas he will observe natural occurrences in real life. In a laboratory experiment it will often be possible to control subsidiary variables: in a real life experiment one is more often faced with the problem of eliminating them in the analysis. It is easy to find exceptions to these tendencies. In experiments on aspects of human behaviour, for example, it is not always possible to control variables as the researcher would wish, even within the laboratory. Human cognitive activity is a substantial component of information retrieval, and in another chapter of this volume, Keen has discussed the problems of laboratory experimentation on that aspect of the field. In contrast, this chapter will consider laboratory work on the mechanical components of information retrieval. The events that the experimenter observes, in this case, are all executed by a machine whose characteristics can, to a large extent, be determined by the experimenter himself. Thus, in principle, he can exercise a considerable degree of control over variables. Indeed, it is hard to thinkof a `purer' laboratory environment in science. The most substantial examples of the type of work to which I refer are the long series of tests with the Smart system (see Chapter 15) and those conducted over a period of several years at Cambridge University by Karen Sparck Jones, C. J. van Rijsbergen and others. 156 I I