IRE Evaluation within the enviornment of an operating information service chapter F. Wilfrid Lancaster Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 126 Evaluation within the environment of an operating information service situation, however, is not so gloomy as it first appears. The local information service has substantial control only over its approach to user-system interaction and the construction of search strategies, but these are in some ways the most important factors to control. They are most important because they are the factors that occur first in the whole operation. If a user's request statement is an inadequate representation of his real information need or if a search strategy is a very imperfect representation of a request statement, the search is virtually doomed to failure. In this case, it matters little whether the indexing is sufficiently exhaustive, whether the vocabulary is sufficiently specific, whether the query language is sufficiently flexible, and so on. Vocabulary, indexing, and other database characteristics may be close to perfect, but this does not help if the search strategies are seeking unwanted information. There is a second reason why control over the processes of user-system interaction and search strategy construction can be considered particularly important. Changes made to these operations can take immediate effect. If today we introduce improved methods of interacting with our users or of constructing strategies, we can expect that the overall effectiveness of our services will improve immediately. But changes to a database tend to produce long-term rather than immediate effects, at least for the retrospective search situation. Changes made to indexing policy, or to the index language, are not going to have a very pronounced effect for some time. Consider a database of 500000 documents growing at the rate of 100000 items per year. Even if sweeping changes were now made to the vocabulary or indexing policies, it would be another five years before these changes would affect even half the total database. Occasionally, of course, an information centre may have complete control over the entire situation: it develops its own database, applies its own hardware and software, prepares its own search strategies and interacts directly with its users. Such a situation is rare. An information centre that is completely in control of its own performance is in a fortunate position indeed. References 1. ASHMOLE, R. F. et al. Cost effectiveness of current awareness services in the pharmaceutical industry, Journal of the American Society for Information Science 24, 29-39 (1973) 2. DAvISON, p.s. and MATTHEWS, D. A. R. Assessment of information services, Aslib Proceedings 21, 28[OCRerr]283 (1969) 3. ORR, R. H. et al. Development of methodologic tools for planning and managing library services, Bulletin of the Medical Library Association 56, 235-267 (1968) 4. CROWLEY, T. and CHILDERS, T. Information Service in Pubik Libraries: Two Studies, Scarecrow Press, Metuchen, N.J. (1971) 5. LANCASTER, F. W. Information Retrieval Systems: Characteristics, Testing and Evaluation, 2nd edn, Wiley, New York (1979) 6. MASON, D. PPBS: application to an industrial information and library service, Journal of Librarianship 4, 91-105 (1972) 7. MAGSON, M. S. Techniques for the measurement of cost-benefit in information centres, Aslib Proceedings 25, 164-185 (1973) 8. MARTYN, J. Unintentional duplication of research, New Scientist 21, 338 (1964) 9. MCDONOUGH, A. M. Jnformation Economics and Management Systems, McGraw-Hill, New York (1963) 10. ROSENBERG, K. C. Evaluation of an industrial library: a simple-minded technique, Special Libraries 60, 635-638 (1969) d