IRE Evaluation within the enviornment of an operating information service chapter F. Wilfrid Lancaster Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Levels of evaluation 107 Measures of benefit present great problems in the information environment since it is difficult, if not impossible, to assess the benefit of `information' or `information service' in any tangible (e.g. financial) terms. Studies of benefit tend to be more subjective than objective. They deal with `perceived value' of the service (in the eyes of the user) or attempt to look at some facet of the impact of the service on the user (e.g. its effect on his information seeking behaviour). A cost-effectivenss study, quite obviously, is one that attempts to relate the cost of a service to its level of effectiveness. It looks at both sides of the cost- effectiveness equation. The cost-effectiveness of the service is improved if either: (a) the level of effectiveness is increased while costs are held constant, or (b) the costs are reduced while the level of effectiveness remains constant. Frequently a cost-effectiveness analysis is conducted in order to choose among several competing strategies for implementing some service. A cost-benefit study, similarly, is one that relates the costs of providing the service to the benefits derived from it. Cost-benefit analyses are really attempts to justify the existence of the service; they ask the question `Do the benefits derived from the service exceed, in some sense, the costs associated with providing it?'. Finally, a cost-performance-benefits analysis is one that investigates the entire set of interrelationships among system costs, performance (level of effectiveness) and benefits (Lancaster5). Because benefits are so difficult to pin down, it is difficult to achieve true credibility in a cost-benefit study of an information service. Several attempts have been made, however, with varying degrees of success. Approaches used include those that try to justify the cost of an information service by: (1) Showing that, if an in-house information service did not exist, it would cost the organization more to buy the same level of service from outside sources (e.g. Mason6, Magson7). (2) Proving that costly research may be duplicated if adequate information services are not available or are not used effectively (Martyn8). (3) Showing that information services can improve the quality of decision making or can reduce the professional level of the staff needed to make various types of decisions (McDonough9). (4) Showing that, if an in-house information service were not available, costs to the organization would increase as engineers, scientists or other professionals are forced to spend more of their own time in information- seeking activities with resulting loss in their own productivity (Rosen- berg10, Kramer11, Mueller' 2). (5) Proving that an information service has contributed tangibly to the organization (e.g. by helping to win a contract, by revealing cheaper solutions to research or production problems, or by stimulating the development of new products). It is probably true to say that most managers would like to be able to prove that the services they provide can be justified from a cost-benefit point of view, but the difficulties involved in such a study have discouraged all but a few attempts of this kind. Consequently, evaluations of information services tend to ignore benefit considerations (in effect, taking the benefit as a kind of