IRE Ineffable concepts in information retrieval chapter Nicholas J. Belkin Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Inference chains and operational definitions 51 uncertain ways, to the original ASK idea. The problem now is: how can one test the validity or accuracy of that original cdnstruct? For instance, if one wishes to know if the end representation is an `accurate' reflection of the user's ASK, and tries to determine this by direct question, the response may he dependent upon any one of the assumptions in the entire chain, with no easy way to tell which of them, or which combination, is at issue in the response. In principle, one ought to examine and test the validity of each of the hypotheses made along the way before performing the test as a whole, but in practice this is very unlikely to happen. The more generally appropriate strategy is to attempt to develop a design in which, as in this example, responses can be directed to each confounding factor. With a chain of assumptions the length of this example, which is probably not unusual, the design may become very complex, and the test instrument clumsy. A possible approach is to run a series of tests, each concentrating upon the end product from the point of view of one of the assumptions, using the data derived from each of the series for design of the subsequent members. But in such a case, there will be some assumption or hypothesis that one cannot, or will not be willing to test (in the example, perhaps the idea of knowledge as a structure of concepts and relations), which must then be considered as an integral part of the original construct itself. An example of this type of problem from the text-related concept group is the question of relative informativeness of text representations. This is an obviously important issue in comparative evaluation of techniques for text representation, especially if the text representations are to be used as the basis for relevance judgements or for matching for retrieval. In the formal case, the situation is that the user, or the user's representative, is presented with some description of a text, on the basis of which a probabilistic relevance decision must be made. In such a case, the obvious strategy is to compare judgements of the representations with relevance judgements of the entire documents, within each system, and then to compare the overall results of the competing systems. Belzer30, for instance, has done an experiment of this sort. In this case, informativeness is operationally defined as the capability of the representation to induce a `correct' relevance judgement in the user. There will be problems in such a design with possible interactive effects of the documents and document representations upon relevance judgements made by any individual, but in general the dependent variable can be fairly well isolated from problems of inference chains, as long as the test is only evaluative or comparative. But, if the purpose of the test is explanatory; that is, if one wishes to explain the differences in informativeness between the representations, then concepts of aboutness, meaning and information become important, and the inference chains from the underlying theory to the eventual representation can cause problems. One now must begin to investigate the assumptions, to see how they have influenced the representations, in order to decide whether the intermediate assumptions, or the underlying concept, are the reasons for the performance. In the case of informativeness as applied to matching for retrieval, the situation is much more complicated immediately, for it must require an aboutness or meaning concept for its implementation. That is, informative- ness here means the ability of the representation system to represent both