IRS13 Test Environment chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 1-12 The test procedure that is followed requires that no more than one single describable change be made to these procedures at any one time, so that search results may be obtained each time one system component is altered. In this way a series of comparisons based on differences in document input may be made, and then perhaps a second series which compares different content analysis procedures. The primary use of the different test en- vironments is to find out whether a conclusion drawn from an experiment in one envit:onment also holds for another. Thus, if a given content analysis procedure is found to be very effective for the computer science collection, a parallel test of content analysis procedures can then be made with the aerodynamics and documentation collections. Conclusions about the effectiveness of search results and system performance generally can be made from different viewpoints using several criteria £33. For various reasons given in (3], the measurement of re- trieval performance oriented towards user satisfaction predominates in the current SMART text experiments, and a discussion of methods and measures used is to be found in Section II of this report. A step recently added to the evalu[OCRerr]ation procedures is that of making statis- tical significance tests of the results, as described in references (4] and [1]. A further[OCRerr]step in the evaluation, which is not formally built into the system requires a hand analysis of the search results involving an examination of individual requests rather than the use only of the averages for a set of requests. A fast-search analysis of specific instances of poor retrieval, for example, is necessary in order to make improvements to the system and to identify areas in which further