<DOC>
<DOCNO> IRE </DOCNO>
<TITLE> Information Retrieval Experiment </TITLE>
<SUBTITLE> The Smart environment for retrieval system evaluation-advantages and problem areas </SUBTITLE>
<TYPE> chapter </TYPE>
<PAGE CHAPTER="15" NUMBER="327">
<AUTHOR1> Gerard Salton </AUTHOR1>
<PUBLISHER> Butterworth & Company </PUBLISHER>
<EDITOR1> Karen Sparck Jones </EDITOR1>
<COPYRIGHT MTH="" DAY="" YEAR="1981" BY="Butterworth & Company">
All rights reserved. No part of this publication may be reproduced
or transmitted in any form or by any means, including photocopying
and recording, without the written permission of the copyright holder,
application for which should be addressed to the Publishers. Such
written permission must also be obtained before any part of this
publication is stored in a retrieval system of any nature.
</COPYRIGHT>
<BODY>
Concluding remarks 327
in a morass of one's own creation. If the system is misdesigned and does not
adequately reflect any part of the real world, the evaluation results themselves
will likely prove to be useless. This point of view has been espoused most
cogently by L. B. Doyle in an early book review44:
`A comment is needed about the Smart system... The word "system" is
misleading. It is really a chemistry laboratory for retrieval principles and
procedures... it is a tour deforce in experimentation in the documentation
area, the like of which is seldom seen... My only reservation about Smart
is that it may not be doing the right kind of chemistry-but then hardly
anyone is... The aspect adjudged most negative is that so much research
should have been done by one party under a suboptimized set of
assumptions....
It is now unfortunately too late to ask the author of the foregoing quote to
explain these statements-the only comment actually made by Doyle raises
the question of `what good is a retrieval system when nine-tenths of the
possible users use the telephone instead ?`-a statement that is surely less
appropriate in 1979 than it was when written in 1969. But obviously the
reviewer's principal contention is certainly correct: if the assumptions in an
evaluation system are suboptimized, the results may not be worth a great
deal.
How then do the Smart assumptions relate to reality? In principle, many
questions can be raised about the appropriateness of the basic model, quite
apart from the problem specifically due to the restricted experimental
environment. Thus the vector space model may be questioned based on the
fact that the scope (as opposed to the subject area) of an item cannot be
represented by a simple vector length and direction. In particular, two items
might cover the same subjects and hence be represented by identical vectors,
yet the topic areas could be treated narrowly in one case and broadly in the
other. The suggestion is therefore made that bibliographic items should be
represented by vectors supplemented by scope or extension measures, instead
of by vectors alone as in Smart45
Other problems, already mentioned in part, concern the implicit
assumptions of term independence in the vector processing model, that is,
the partially false notion that content identifiers occur independently of each
other in document and query vectors46. Independence among evaluation
parameters is also assumed by certain statistical tests used in Smart to assess
the significance of the evaluation results. Additional methodological
objections are easy to find in a computer environment comprising many tens
of thousands of processing steps.
Some of these comments are formally correct; a laboratory model by its
very nature can never fully reflect the real life conditions. The question is
whether the deviations are sufficiently serious to affect the usefulness of the
model. So far as Smart is concerned, the first-order characteristics of the real
world are believed to be properly represented-the subject content of a
document represented by a vector in multi-dimensional space is more
important than its extension, and to a first-order approximation, the terms
used to characterize the documents are indeed independent. Furthermore,
while the formal proofs of effectiveness of some Smart procedures are
applicable only in restricted environments (binary vectors, term independ-
</BODY>
</PAGE>
</DOC>