IRE Ineffable concepts in information retrieval chapter Nicholas J. Belkin Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Definitions or interpretations of the concepts 45 (1) User-dependent concepts: information need; desire. (2) Text-dependent concepts: information; aboutness; meaning. (3) User and text-confounded concepts: satisfaction; effectiveness; synthema (homeosemy). This categorization of issues derives from the general structure of information retrieval systems2, in which documents and needs are each separately represented, then matched against one another in order to retrieve documents which are judged by the user according to their appropriateness to her/his need. This situation requires concepts basic to need representation and need understanding, concepts basic to text representation and under- standing, and concepts concerned with the relationships between text and need. Various concepts basic to each of the three areas outlined above have been widely discussed in the literature, although not always to great effect as far as testing of information retrieval systems is concerned. Usually, the test of any system has been concerned with secondary entities or processes in one of the sub-areas (such as comparative experiments on indexing systems), stopping short of investigating the relationship of underlying concepts (such is information or aboutness) to the results of the tests, or even of determining whether there were any such underlying concepts to the systems being tested. The suggestion here is that it may now be the right time to begin such investigations, to make these concepts at least explicit in testing, and perhaps even to make them the basic variables in the testing of information retrieval systems. Before continuing this argument, some general discussion of these concepts themselves is in order. 4.2 Definitions or interpretations of the concepts Jser-related concepts Although this group of concepts seems to be the obviously central core to the information retrieval situation, since evaluation of system performance should be solely in its terms3'4, it seems to be the most neglected in the literature of information retrieval system testing. This may be because concepts such as relevance, which depend upon this group but are confounded with the text-related concepts, have been initially more important to systems testers in that they provide the means for direct comment on system l)erformance. There has been somewhat more treatment of user-related concepts in such areas as reference work5 and in theoretical discussions of 4 information science The basic situation, as outlined by Taylor5, is that of a person coming to an information system with some already (at least vaguely) recognized need, md going through various stages of representation of that need which culminate in a formal request put to the information retrieval mechanism in terms which it can use for matching against its store of texts. In this situation, one can recognize a number of elements which are likely to affect the iiiechanism in significant ways, yet which are difficult to describe or quantify. The first of these is the desire of the user. This concept seems not to have l'een discussed explicitly as a separate issue, but one should note that, apart t#()m a need for information, the user comes to the mechanism with some set