<DOC>
<DOCNO> IRE </DOCNO>
<TITLE> Information Retrieval Experiment </TITLE>
<SUBTITLE> Opportunities for testing with online systems </SUBTITLE>
<TYPE> chapter </TYPE>
<PAGE CHAPTER="7" NUMBER="129">
<AUTHOR1> Elizabeth D. Barraclough </AUTHOR1>
<PUBLISHER> Butterworth & Company </PUBLISHER>
<EDITOR1> Karen Sparck Jones </EDITOR1>
<COPYRIGHT MTH="" DAY="" YEAR="1981" BY="Butterworth & Company">
All rights reserved. No part of this publication may be reproduced
or transmitted in any form or by any means, including photocopying
and recording, without the written permission of the copyright holder,
application for which should be addressed to the Publishers. Such
written permission must also be obtained before any part of this
publication is stored in a retrieval system of any nature.
</COPYRIGHT>
<BODY>
Constraints on real life testing 129
An attempt will be made to define the environment in which one is
experimenting; to point out the restrictions that are imposed by such an
environment; and to demonstrate the opportunities that undoubtedly exist
for evaluation and subsequent improvement of working systems.
7.2 The environment
The systems that are increasingly important, and therefore warrant attention,
are the online bibliographical retrieval systems such as DIALOG, ORBIT,
INFOLINE, etc. These systems are normally used from a terminal and with
interaction from the user or intermediary, but include in their function the
facilities previously available in a batch oriented computer system. The
online systems, however, offer many more opportunities for analysis and
provide the possibility of performing better searches as well as the pitfall of
unknowingly doing very poor searches.
The essential function of any information retrieval system is to satisfy a
need for bibliographical information as expressed by the user. The advantage
of an online system over a batch system accessing the same database is that
it allows the user to modify the expression of his requirement in the light of
information provided by the system. We ought to expect that in these
circumstances users would achieve better results from an online system.
Tests in such an environment should aim to prove or disprove this belief.
From the results of such investigations one would hope to determine methods
of improving the system and gain a greater insight into the real needs of
users.
7.3 Constraints on real life testing
The tests that need to be applied to any information retrieval system are
concerned with determining whether a client's information need is satisfied
by the system. Both the need and the satisfaction are more difficult to define
in an online system than in a batch system. The starting point for either
system is the client, who may have a written question or merely an idea in his
head. This question is translated into a search statement generally with the
help of an intermediary. For a batch system this statement is deemed to be
the definition of the user's need; with an online system this is not necessarily
the case since the user can, and should, change his search statement in the
light of information supplied by the system. The difficulty is to decide which,
if any, of the search statements used actually defines the user's need, and in
particular whether the need as opposed to the definition of it, has changed
during the search process.
Satisfaction is equally difficult to define. A user of an online system can be
satisfied in a number of ways. The client requiring all references on a topic
obviously needs high recall, and is prepared to tolerate some reduction in
precision. Such clients are similar to former batch system users, who, because
of the intrinsic delay in the early systems, only undertook literature searches
when they required a full bibliography.
Online systems with immedate access to the database also provide for the
</BODY>
</PAGE>
</DOC>