ISR10 Evaluation of Document Retrieval Systems chapter Joseph John Rocchio Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 5-18 evaluatich experiments for retrieval systems. Conventionally, on an experimental basis, there is a serious problem in obtaining representa-' tive search requests with accompanying relevance judgments (witnessed by the controversy as to the value of the test queries used in the 10 Cranfield study for example). [OCRerr]ather than measuring performance by the use of test queries, a retrieval system can be evaluated by obtaining user judgments as to the degree of association between documents which by their location in the index space are necessarily associated. Such an evaluation represents a measure of the ability of the indexin[OCRerr] [OCRerr]cheme to preserve the associations in the index space which users can detect from the information content as expressed in the natural langnage. Since the retrieval performance which a user can expect is a function of the degree to which input search requests correspond (or can be, adjusted to correspond) to `their respective optimal forms, and since the performance of an optimal query is directly related to the consistency of the associations of documents in the' index space, a measure of the latter is indicative of a measure of the former. While such a procedure does not mecessarily reduce the quantity of subjective data required. for a significant test of a retrieval system, it offers the potential for providing an increased measure of experimental control Since the requirements for examining the implications of such a test program are prohibitive with respect to the scope of this thesis.(by virtue of requiring a large subjective test effort), this test de,sign is offered as a suggestion worthy of additional consideration.