IRE The pragmatics of information retrieval experimentation chapter Jean M. Tague Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Decision 3: How to operationalize the variables? 65 printed indexes, card catalogues, and other manual files and in online interactive computer searching, the response to one search statement triggers the next, until the searcher is satisfied. The length of this process can be measured by the total number of search statements or the search time. These measures, however, confound the complexity of the search and the experience or style of the searcher. Variables describing overall complexity as a function of inter-statement relationships need to be developed. Another search process variable is the form of the output: citation, citation and abstract, citation and index terms, citation and full text, etc. Particularly in online systems, there may be a choice of output format. In fact, in online searching the term output is ambiguous. Typically, one keys in a boolean search statement and gets as response a statement of the number of hits. If this is large, no further output at all may be requested, or a few titles only may be scanned. The strategy may then branch in one of several directions: list all citations online modify the search statement list all citations offline terminate the search In an experimental situation, a choice must be made as to what constitutes output. No consensus appears to have yet been reached in the field. This is unfortunate, as measures of effectiveness obviously depend on how output is defined. Marcus, Kugel and Benenfeld5 have introduced the idea of the indicativity of an output field. It is defined operationally as the proportion of documents which the field indicates to be relevant which are actually assessed as relevant from full text. It appears that indicativity is related logarithmically to the length (i.e. number of words) in the field. PeopI[OCRerr]indexers, searchers, users Because many of the decisions in information retrieval are based on human judgement, the professional training and experience of the system personnel and of the users is often studied. These variables can usually be operationalized in terms of years of experience and/or training, in general or with particular Systems. Job titles are an unreliable guide, as they vary from installation to installation. Number of previous searches can also be used as a measure of experience with a particular system. Many of the things one wants to know about people can be characterized as attitudinal variables: ease with other people, acceptance of automated processing. Experimenters should be aware of the large psychological literature on attitude measurement, and consider the use of one of the standard scales[OCRerr]Guttman, Lickert, Thurstone, the `unfolding' method. A good text is Lemon6. He defines attitude measurement as gathering observations about people's behaviour and allocating numbers to these observations according to certain rules. Attitude scales depend on the investigator's theoretical assumptions about the nature of the attitude he or she is trying to measure, its relationship with behaviour, and the rules used