IRE Ineffable concepts in information retrieval chapter Nicholas J. Belkin Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 54 Ineffable concepts in information retrieval their own right; the second is as possible sources of error or variation in other aspects of the system with which the tester is primarily concerned. Each of these situations raises different questions and different problems of interpretation and control. The major problem in the first case lies in thc construction of operational definitions of the variables to be studied. This may be difficult, but once these definitions have been established, it may be possible to study the variables in isolation. For instance, if one wants to investigate the concept of `aboutness' experimentally, one could control the experiment to include, say, only documents and readers of documents, without reference to an information retrieval system at all. The problems of the second case, on the other hand, arise because the interactions of variables within information retrieval systems are so complex. In this situation, it is in general not possible to isolate variables completely. For instance, if one wants to evaluate some method of content description within an information retrieval system, then the relationships between the concept of aboutness, the description method and the evaluation measures are significant, even though aboutness is not an explicit variable in the experimental paradigm. Thus, in the second case, which is likely to be the usual situation in information retrieval system testing, one ought to begin by attempting to isolate all of these variables (conceptually), and then to see to what extent they actually might influence the variables in which one is interested, and the evaluation measures. In this case, again, one is not interested in the `ineffable' variable itself, but rather in the effects that not having included it in the design might have on the results of the test. It is useful here to discuss, in rather general terms, the extent to which being able to deal with these concepts might affect an information retrieval test. Text-related concepts It is possible that having a specific, operationally definable and experimentally tractable information concept is a necessity for the development of theory in information retrieval. Nevertheless, there is some question as to whether this particular concept need be made explicit in evaluation tests, at least of existing systems. The reason is that there is usually only a minimal relationship between whatever passed for an information concept in the system as originally formed and the measures used to evaluate the system's performance. On the other hand, if one wishes not only to compare, but also to explain differences between systems, then both information and aboutness become quite important. This is especially true if the focus of attention is the description mechanism, rather than, say, the retrieval strategy. If one wishes seriously to explain the difference between two description mechanisms, then one must be able to discuss the relationship of each mechanism to an underlying information or aboutness concept. For instance, one could compare two systems of automatic indexing in terms of retrieval performance (recall and precision, say) and discover that one of the systems gave consistently better results. But to be able to say why this was the case, one would need to consider the underlying assumptions of each method, in terms of an underlying information or aboutness concept. One could then make some decisions about whether the difference in performance lay in a different technique deriving from the same assumptions, or in different assumptions. I I I I I I I i I i I I I I I I- h I I I I 5 I I