IRE Retrieval system tests 1958-1978 chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 1 12 Retrieval system tests 195[OCRerr]1978 Karen Sparck Jones Previous chapters considered the problems of information retrieval experi- ment in general, and particular types of experiment. This chapter looks at information retrieval experiment over the last twenty years as a whole, to see what the actual tests which have been carried out show. I shall not attempt an exhaustive review of this work. I shall seek rather to characterize it by referring on the one hand to especially significant tests, and on the other to average or representative ones. My object is to exhibit the development of retrieval experiment in the last two decades in terms of the purpose, quality and influence of the tests which have been carried out. The twenty year period for the survey is a natural one, since it effectively covers the development of modern, especially computer-based, retrieval systems, and equally, most of the significant information retrieval tests. 12.1 Experiment and investigation The amount of work done under the general heading of information system studies is very large. To see the wood in the trees it is essential to have a clear view of what constitutes an experiment and to restrict the survey, as far as possible, to experiments in the strict sense. Thus for the purposes of this chapter an experiment is distinguished from an investigation in the following ways. An experiment aims at explanation, an investigation only at description: an experiment seeks to answer questions about what happens if such and such is done, by showing why it happens; an investigation indicates only what happens. In the context of information retrieval system testing, an experiment typically focuses on individual variables, where an investigation exhibits system behaviour as a whole. An experiment is in principle hypothesis-guided, while an investigation may be no more than hypothesis- generating. The key requirement of experiment is therefore control over test variables, both primary and secondary. In consequence, experiment is concerned with measurement. Investigation may also produce measurments, and in both experiment and investigation measurement may be merely descriptive. However since an information retrieval system has a function, any measurements must ultimately be related to system performance in terms 213