IRE Opportunities for testing with online systems chapter Elizabeth D. Barraclough Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 130 Opportunities for testing with online systems user who wants to gain an entry into a subject by a `quick and dirty' search done at low cost. User satisfaction is almost guaranteed for these clients provided one relevant reference is found within a reasonable time. At the other extreme are clients who want to ensure that their chosen area of research is untouched, i.e. they want to find that there are no references in the literature. The situation facing the experimenter who wishes to carry out tests in an online environment is thus quite complex. The users can come with a variety of needs; the definition of these needs is not precise in every case; measures of satisfaction depend upon the type of need; and performance of the system is also related to the expressed need. An example of the difficulty, or perhaps futility, of attempting to measure satisfaction is shown by Lawton, Auster and To2. However, one should not despair: an online system attracts a large number of users, who, by appropriate methods, can be classified. These methods will rely to a large extent on interviews with the user both before and after the search is undertaken but an attempt should also be made to classify users' needs automatically using parameters such as: search complexity, e.g. number of conjunctions in a search statement; number of references printed; and number of references deemed to be relevant. Similarly, the individual user's need would have to be elicited by interview but should be checked against the various search statements attempted. Producing such a classification of types of requirements and methods of assessing the user's needs is a subject worthy of study on its own and needs to be undertaken before extensive tests can be carried out. 7.4 User constraints Given that parametrization of the users and their needs is possible, the only major constraint on the user sample is the reluctance or inability of some users to take part in an experiment because of the confidential nature of their research. To exclude such users completely could introduce a significant bias into the results. Users from industry could have a rather different requirement from the system which could be neglected if they were unable to take part. Most users are very willing to take part in experiments particularly if the cost of the service is reduced (preferably to zero) for participants. Despite all these problems it should prove possible to select various groups of users to test the ability of the system to supply a variety of user needs. Both the function of the system considered as a whole and the facilities provided within different parts of the system, e.g. for search formulation, should be investigated. 7.5 System and database constraints Real life systems impose particular constraints on experimenters which are associated with changes over time. Working systems do not stand still; the data is continuously changing as new information is added to the database. As the files grow the database is split and only the most recent part is kept in I