IRE Evaluation within the enviornment of an operating information service chapter F. Wilfrid Lancaster Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 122 Evaluation within the environment of an operating information service `behavioural intervention' are recognized in the design. The study was conducted in a research and development centre of an industrial organization. Level 1 of the technical information intervention was a conventional SDI service designed to support the work of selected project teams. Level 2 was a more personalized service in which technical information staff attended regular meetings of project teams and developed targeted information `packages' to support the work of the teams. The behavioural interventions represented deliberate attempts to enhance the existing channels of formal and informal communication within the company. The basic hypothesis was that improved productivity within the centre would be more likely to occur through the interaction of both types of intervention-that improving the formal information service was not in itself enough to achieve improved acquisition, use and transfer of information. As the design indicates, some project teams received the highest level of behavioural intervention and the highest level of information service intervention, others received various combinations of the levels of intervention, and the control group received nothing in addition to the regular services offered by the technical information centre. This is a highly complex design that presents great problems in implementation. As Olson points out: `Substantial difficulties were encountered. We had underestimated the difficulties of making the role changes in the information centre. In spite of the full support of the information centre manager and the directors of research and an agreement by the staff to carry out the interventions, there was substantial reluctance. The information centre staff saw some risks to beginning an experimental process which would change the existing patterns of information flow/non-flow. We succeeded in dealing with the staff's concerns by securing additional temporary help for the centre, by assisting the staff in developing information profiles, by top-level management assurance to the staff that their full participation in the project was highly valued, by meeting with the scientists and engineers to prepare them for the new role of the information centre, and by followup coaching and support meetings as necessary.' A useful distinction has been made, by Giuliano and Jones24, between proof-oriented and insight-oriented studies. Experimental research is con- cerned with proof and rejects insight. But the manager of an operating service may be satisfied with much less than the proving of a hypothesis within acceptable limits of statistical confidence. He may be perfectly satisfied with gathering enough evidence to give him insight into a situation and be willing to make decisions on the insight alone. It is frequently possible to gain useful insights with very small studies. Much larger and more expensive studies would be needed to provide `proof'. Moreover, proving that option A performs better than option B does not in itself explain why A is better. Even with `proof', the manager of a centre may not be willing to adopt A, or change to it, unless he understands the reasons why it appears to give better results, and so may still rely on some measure of interpretation or insight in making a particular decision affecting the future of the system. Giuliano and Jones discuss the distinction in the following terms: `One decision which has to be made is whether the effort is to be put into I