IRS13
Scientific Report No. IRS-13 Information Storage and Retrieval
Test Environment
chapter
E. M. Keen
Harvard University
Gerard Salton
Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government.
I-li
to real user ones, contain more instances of relevance decisions that might
be disputed than the other collections. It is suggested that real users tend
to have less clearly defined requests in mind, and tend also to judge relevance
by means of requirements that they fail explicitly to state in the request.
The validity of prepared requests and relevance decisions for experimental
testing is frequently challenged by opinion, but a controlled experiment that
will show the differences (if any) for test purposes between prepared and real
requests is still not at hand. Studies of agreement between different judges
carrying out an identical relevance decision task have shown that poor agree-
ment frequently results. But a more important question for experimental tests
is wh-'.ther differences in relevance decision actually alter comparative test
results; that is, does option one perform better than option two both when
person A does the relevance decision, and person B, and also when relevance
decisions of both persons, or those common to both are used? A new docu-
mentation collection known as the ISPRA/Euratom collection is being used to
test just this problem; test results will appear in a future report in this
series.
4. Text Experiments
A) Experimental Procedures
The laboratory environments that have been described permit controlled
tests by meansof simulated searches. The operation of a retrieval system
may be separated into three stages: input of the documents and requests to
the system, procedures of content analysis applied to documents and requests,
and the matching of the requests with the documents which is the output stage.