IRE Ineffable concepts in information retrieval chapter Nicholas J. Belkin Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 48 Ineffable concepts in information retrieval by the user of the appropriateness of text to need based not upon the question put to the system but rather on the user's entire desire and need state at the time of receiving the text for judgement. Each of these concepts of relevance is obviously quite different from the others, and each has special advantages and problems in testing situations. The major point here is that the choice of relevance concept will radically affect evaluation of system performance. The most commonly used measures of performance, recall and precision, depend solely on relevance judgements, and thus are peculiarly sensitive to this concept. Utility, the other major proposal for performance evaluation (see, e.g. Cooper25), can in principle take account of other factors than relevance in the destination's or logical senses, but at the moment practical difficulties in assessing this measure make its use difficult. Karlgren26, with homeosemy, and Robertson27, with synthema, have independently attempted to formalize the topic relationship between text and question. They both suggest that the problem may be considered as the degree to which the aboutness of each coincides with that of the other, and that this relationship, especially in Robertson's suggestion, might provide a scale which would underlie the concept of relevance. Notice the extent to which both of these notions depend upon the concepts of aboutness or meaning. These suggestions provide a potentially valuable way of partitioning the various aspects of the relevance concept, but as yet are still only suggestions, with no concrete methods of implementation. To return to the general notion of satisfaction, one can see that logical and destination's relevance, and more formally synthema or homeosemy, refer only to conceptual aspects of the need, and assume that the question is an adequate representation of the need. Situational relevance extends the notion to include beliefs and other aspects of the user's condition, while accepting the possibility of adequate linguistic representation, but pertinence and utility go even farther in attempting to take into account affective and other aspects of the need, and do not necessarily assume that the question expresses the need in its entirety. Usefulness is another term that has been used in place of relevance in attempts to include more than strictly topic-related aspects of need in the judgement process. Apart from satisfaction, in terms of relevance or utility, effectiveness appears to be the only major candidate as a basis for information retrieval system evaluation. If, for example, information is considered as `data of value in decision making28', then system performance can be evaluated by whether its output was used in making appropriate decisions; that is, by some `objective' or at least behavioural measure of the effect of its output on the user. This approach completely bypasses the question of relevance, but is unfortunately very difficult to operationalize, especially in obtaining and measuring observations of the effect. It also poses significant difficulties in prediction of effect, because of its situation dependence. 4.3 Interactions The concepts discussed in this chapter are difficult to deal with in an experimental or investigatory setting not only because they are conceptually and practically not very tractable, but also because they are highly