IRE Ineffable concepts in information retrieval chapter Nicholas J. Belkin Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Definitions or interpretations of the concepts 47 with a text. This makes ferreting out that concept, or making it explicit, of potential importance in evaluation of such schemes. The concepts of meaning and aboutness are also central in the question of text representation, presenting similar difficulties of interpretation. In general, information retrieval has been concerned with what a text is about (I' C. some topic specification) rather than what it means (e.g. the propositional [OCRerr]tructure of the text). But just what the relationship between these two concepts is, and how one interprets them, is still a significant problem. Some [OCRerr]tescribe aboutness as a whole-text phenomenon1 5, others as a phenomenon related to the state of knowledge of the reader16. The two positions are quite Ilitferent in their predictions and in their prescriptions concerning text (lescription systems, which makes it important when evaluating them to know on what premises the system has been based. Robertson' 7 and Sparck lofles'8 have recently provided good discussions of the relationship between Illeaning and aboutness in the information retrieval context. The major [OCRerr] lect that such problems might have on information retrieval system testing [OCRerr] that different representation systems might be based on different concepts (It (boutness, or might even be based on a concept of meaning (as, for ii[OCRerr]tance, one assumes the LEADERMART system is' 9), and thus differences ii performance might be explainable only in terms of these underlying ([OCRerr]()I[OCRerr]cepts. Certainly, as Gardin20 demonstrates, systems based on meaning \[OCRerr]IIJ be substantially different from those based on aboutness. ([OCRerr]()ll(ounded concepts I lii[OCRerr] group of concepts is concerned with the relationship between need and I([OCRerr][OCRerr]t, and can be most generally described as having to do with satisfaction' (with two major exceptions). This group, or at least some aspects of it, is the Ilile of the three discussed which has been most thoroughly investigated in l([OCRerr] context of information retrieval system testing, for the good reason that It provides the basis for evaluation of information retrieval system Ileltormance. *l[OCRerr]he best known of these concepts is relevance. Relevance, in its most [OCRerr]([OCRerr]!)eral information retrieval sense, describes the appropriateness of a text to `.;)ecific information need. Saracevic21 has written a fine review of research .111(1 speculation on this concept in information science, but we can mention t[OCRerr]cie a few of its problems and a few proposals for dealing with them. A major ililliculty in information retrieval system testing has been to obtain reliable .[OCRerr]i([OCRerr]t reproducible relevance judgements. The problem has been that the Ilitilnate judge of relevance in the real world is the person with the lIllIbrination need, and this person's formal question to the information I([OCRerr]([ieval mechanism (which is all that mechanism has to work with) may not ([OCRerr]l((()mpass all of the factors which bear upon her/his eventual relevance I(((t(',cment. So relevance as a general concept has tended to be divided into WI or three separate concepts (see, e.g. Kemp22 or Wilson23). These include: lI,gical relevance24, which requires that the propositions of the request be lI(vIiI(tCd in, or logically deducible from the text; destination's relevance or (I([OCRerr].I relevance, which depends only upon the relationship between the topics iii the t'ormal question and the texts retrieved, usually as evaluated by an I.[OCRerr] t([OCRerr]rfl1l judge, and situational relevance or pertinence, which is the judgement