IRE Introduction chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Introduction 5 We can think of experiments as suggesting methods of investigation, and investigations as influencing the design of experiments. Experiment is typically hypothesis-guided, investigation hypothesis[OCRerr]generating. However we must allow that the hypothesis underlying an experiment may be implicit rather than explicit, and, further, may be of the weakest kind, namely `This variable is important and therefore worth study' But this difference between experiment and investigation is not the significant one: the essential difference between experiment and investigation is in the application of control in experiment. One cannot test without control. The problem in experiments is then to control variables and not to suppress them. Experiment implies, and investigation typically involves, measurement and specifically, since an information retrieval system has a purpose, measurement related to system merit in some sense. However investigations may be summed up in measurements of a descriptive rather than evaluative kind. Moreover both experiments and investigations may be concerned with individual system elements only, and with direct measurements of these, so performance measurement proper is indirect rather than direct. Finally, although experiment and investigation may in principle refer equally to laboratory or operational system studies, in practice there have been few operational system experiments, and experiment and investigation tend to refer to laboratory and operational studies respectively. In practice, this distinction between experiment and investigation may be difficult to apply to work that has been done in information retrieval, particularly in relation to operational systems. Thus while focusing on experiment, the book does include discussions of better-conducted and more systematic investigations. Equally, many of the points applicable to experiment are applicable to investigation. The word `test' is therefore used as a global, neutral term to cover both experiment and investigation, and also as a stylistic variant to refer to either when the context makes the particular interpretation clear.