IRE Ineffable concepts in information retrieval chapter Nicholas J. Belkin Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. The significance of `ineffable' concepts in information retrieval testing 53 issumptions about the independence of index terms, and about the nature of the space or scale in which the entities are to be compared. Only after all of these three types of assumptions (text-related, need-related "md matching-related) have been granted, does one actually achieve the original goal; to establish some operational means of determining the tiomeosemy of a document with a need. And just as in the ASK example, (`Ich of these assumptions has strong theoretical implications which ought. iii principle, to be tested. The point of this discussion has not been to discourage investigation of these complex and basic concepts, but rather to indicate the sorts of &lifficulties one can expect in trying to deal with them, and to make some tentative suggestions about how one might deal with them. These suggestions can be summarized as: first, make certain that all of your assumptions or hypotheses have been made explicit; secondly, try to minimize the steps in the chain from theoretical construct to operational definition; thirdly, design the test to investigate, as much as possible, the effect of each assumption; and finally, be explicit in reporting the decisions about those assumptions which were left untested. With concepts such as need, information and satisfaction, such chains will always be necessary. This does not mean that the concepts cannot be studied and included in tests, but it does mean that such tests must he unusually self-conscious in their design. There still remains the problem of generality of theory. In the first example of this section, there was a minimal theory that ASKs underlie information needs, and furthermore that ASKs can be represented as certain types of structures. Now from such theoretical statements one can indeed generate some predictions or procedures for making them operational, but it is quite difficult to construct these so that their inadequacies or failures can be interpreted as invalidations or falsifications of the theory itself. In the ASK example, people might be asked to comment upon the relative accuracy of the representation of their need, but negative comments might have no bearing upon whether the original theory is valid, for the elicitation technique or the specific representational format might be equally at fault. Similarly, one might be able to predict that certain types of `anomalies' might be associated with certain classes of information need, but if the prediction fails, it can be interpreted as doing so only at the representational or classificatory levels. There seems no simple way to avoid this sort of problem with these concepts, so that perhaps what one should do is to accept it, and to consider these concepts as basic assumptions which lead to particular strategies or systems for solving certain operational problems. In this way, one can evaluate a system as a whole (but taking account of inference chains) according to how well it solves the problems (or achieves its goals). This could be done in comparative or single evaluative contexts, and one would attempt to judge the theory not according to absolute validity, but rather according to how well the framework which it establishes works in the context of the problems it has been constructed to solve. 4.5 The significance of `ineffable' concepts in information retrieval testing There are two ways in which the variables discussed in this chapter are important in information retrieval testing: the first is as objects of study in