IRE Opportunities for testing with online systems chapter Elizabeth D. Barraclough Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. The opportunities of real life Systems 131 ilic current file. Thus any recall and precision assessments must be done at I lic s[OCRerr]ime time to ensure that the same part of the database is used otherwise I[OCRerr]iiI1)arisons are not valid. System providers can also add new databases to flicir files and hence extend the search possibilities; again, by making [OCRerr]`[OCRerr]ii11parisons at the same time, this problem can be avoided. Iii vestigation of the facilities provided by the system can be affected by Ii[OCRerr]inges that the system providers may make over time. Thus they could offer hiore facilities for inspecting the dictionary and any statistics collected over %iich a change would be invalid. Perhaps the most serious constraint imposed by the system on the user and hence on the test possibilities, is the time pressure caused by the method of charging. Many users, or their librarian intermediaries, are very conscious of tuc cost and therefore look for a quick solution rather than taking time to consider how to achieve a better result. The system providers are in some (Iirnculty here; they have to implement a charging mechanism which covers both the cost of running the system and the cost of creating and making 4ivailable the databases. They are also constrained by the facilities that are provided by the machine manufacturers for implementing a charging scheme. As a result of these constraints most systems base their charges on (he time the user is connected to the system and the number of references retrieved and printed. The second parameter is probably quite justifiable. The first parameter also has benefits for the system provider as it encourages users to access the system at slack times as an overloaded system will run more slowly and thus be more expensive. From the user's point of view, this method of charging is most unfortunate. He is charged for thinking time, term selection and searching in the database at precisely the same rate. He ought, at least, to be able to think `freely'. Also many facilities that only require access to the dictionary are not costly in computer time or storage, but would, if provided cheaply, offer the means of formulating much better searches. For users who are accessing the systems from remote locations, particularly in the UK, the charging is confounded by the fact that telecommunications costs also have a component which is duration dependent in addition to the charges related to the volume of data. The effect of this crude method of charging for system use is to force users to take a gross view of the system rather than considering the effectiveness of different parts of the system. This naturally imposes the same limitations on the tests that can be attempted, unless the constraint can be removed by providing free searches or a local charging mechanism. 7.6 The opportunities of real life Systems The advantages of testing in a real environment, for many purposes, outweigh the constraints that they impose. The most immediate advantage is the possibility that tests carried out with real systems can bring direct benefit to the users in a realistic time-scale as opposed to testing on static systems where the application of the results is not immediately obvious. The feature of online systems, which cannot be overestimated for testing and evaluation, is the ability to observe user behaviour. This can be done in