IRS13 Test Environment chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 1-28 6. Request and Collection Comparisons Some of the test environment characteristics presented in parts 2 and 3 and Figs. 1, 3, and 5, may be evaluated using search performance re- suits. Three types of comparisons are made ,in the next few paragraphs. A) Request preparation The set of 34 requests available for the IRE-3 computer science collection is made up from two sets of 17, each set being distinguished mainly be the persons preparing the requests and making the relevance decisions. A comparison of retrieval performance of the request set prepared by staff members with the set prepared by a non-staff person is made in Fig. 12. Fallout is used because average generality on the staff set is 27.5, with a mean of 21.6 relevant per request1 and on the non-staff set, generality is 16.8, with a mean of 13.2 relevant per request. This suggests that the non-staff requests are more specific than the staff requests, and re[OCRerr]examination does show that the staff requests are a little longer (see footnote in Fig. 1). Three of the staff requests are found to be very similar to three of the non-staff ones. A comparison of these three pairs is therefore given in Fig. 13 Relevance de[OCRerr]ision agreement is quite strong for two of the pairs, but in every case the staff request is the longer and exhibits a quite superior retrieval performance. The variables to be considered in examining this type of request preparation and.relevance decisions are known to be numerous, and it is not surprising that these subjective tasks have a large effect on the performance outcome. A paper by John O'Connor (13] and work done at System Development Corporation [14] provide further knowledge of these variables which can be used in future