IRS13
Scientific Report No. IRS-13 Information Storage and Retrieval
Test Environment
chapter
E. M. Keen
Harvard University
Gerard Salton
Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government.
1-28
6. Request and Collection Comparisons
Some of the test environment characteristics presented in parts 2
and 3 and Figs. 1, 3, and 5, may be evaluated using search performance re-
suits. Three types of comparisons are made ,in the next few paragraphs.
A) Request preparation
The set of 34 requests available for the IRE-3 computer science
collection is made up from two sets of 17, each set being distinguished
mainly be the persons preparing the requests and making the relevance
decisions. A comparison of retrieval performance of the request set
prepared by staff members with the set prepared by a non-staff person is
made in Fig. 12. Fallout is used because average generality on the staff
set is 27.5, with a mean of 21.6 relevant per request1 and on the non-staff
set, generality is 16.8, with a mean of 13.2 relevant per request. This
suggests that the non-staff requests are more specific than the staff
requests, and re[OCRerr]examination does show that the staff requests are a little
longer (see footnote in Fig. 1). Three of the staff requests are found to
be very similar to three of the non-staff ones. A comparison of these
three pairs is therefore given in Fig. 13 Relevance de[OCRerr]ision agreement
is quite strong for two of the pairs, but in every case the staff request
is the longer and exhibits a quite superior retrieval performance. The
variables to be considered in examining this type of request preparation
and.relevance decisions are known to be numerous, and it is not surprising
that these subjective tasks have a large effect on the performance outcome.
A paper by John O'Connor (13] and work done at System Development Corporation
[14] provide further knowledge of these variables which can be used in future