IRE Evaluation within the enviornment of an operating information service chapter F. Wilfrid Lancaster Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 110 Evaluation within the environment of an operating information service criteria that seem most important in the information service environment are listed in Table 6.1. Cost factors are as important in the evaluation of information services as they are in the evaluation of other services and products. The service must be provided at a cost that the user feels is reasonable in relation to the benefits associated with it. Cost to the user involves more than direct charges. It includes the cost of his own time, that is, how much effort is involved in the use of the system. Studies of the information-seeking behaviour of scientists and other professionals have consistently shown that accessibility and ease of use are the prime factors influencing the choice of an information source. In general, the most convenient source of information is chosen, whether or not it is perceived by the user to be, in some sense, `the best'. TABLE 6.1. Criteria by which users will evaluate an information service Level 1. Evaluation of effectiveness (considerations of user satisfaction) a. Cost criteria (1) Monetary cost to user (per search, per subscription, per document) (2) Other, less tangible cost considerations (a) Effort involved in learning how to use system (b) Effort involved in actual use (c) Effort involved in retrieving documents (through backup document delivery systems) (d) Form of output provided by the system b. Time criteria (I) Time elapsing from submission of request to retrieval of citations (2) Time elapsing from submission of request to retrieval of documents (3) Other time considerations[OCRerr]for example, waiting time to use an online system c. Quality considerations (1) Coverage of the data base (2) Completeness of output (recall) (3) Relevance of output (precision) (4) Novelty of output (5) Completeness and accuracy of data Level 2. Evaluation of cost effectiveness (user satisfaction related to internal system efficiency and cost considerations) (1) Unit cost per relevant citation retrieved (2) Unit cost per new, that is, previously unknown, relevant citation retrieved (3) Unit cost per relevant document retrieved Level 3. Cost-benefit evaluation (value of system balanced against costs of operating it) Ease of use factors include ease of interrogating the system in the first place, that is, ease of making one's needs known, and ease of use of the output provided by the system, especially the ease with which the output can predict the relevance of the documents it refers to. A very important facet of the latter is availability of an efficient and convenient document delivery capability. A service that stops at the delivery of bibliographic citations goes only part of the way toward satisfying an individual's information needs. Such a service causes considerable frustration if the user is unable to obtain the documents cited or can do so only through procedures that he views as inconvenient and time-consuming. The users of information services have various kinds of information needs1 including the need for: J