IRS13 Test Environment chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. I-li to real user ones, contain more instances of relevance decisions that might be disputed than the other collections. It is suggested that real users tend to have less clearly defined requests in mind, and tend also to judge relevance by means of requirements that they fail explicitly to state in the request. The validity of prepared requests and relevance decisions for experimental testing is frequently challenged by opinion, but a controlled experiment that will show the differences (if any) for test purposes between prepared and real requests is still not at hand. Studies of agreement between different judges carrying out an identical relevance decision task have shown that poor agree- ment frequently results. But a more important question for experimental tests is wh-'.ther differences in relevance decision actually alter comparative test results; that is, does option one perform better than option two both when person A does the relevance decision, and person B, and also when relevance decisions of both persons, or those common to both are used? A new docu- mentation collection known as the ISPRA/Euratom collection is being used to test just this problem; test results will appear in a future report in this series. 4. Text Experiments A) Experimental Procedures The laboratory environments that have been described permit controlled tests by meansof simulated searches. The operation of a retrieval system may be separated into three stages: input of the documents and requests to the system, procedures of content analysis applied to documents and requests, and the matching of the requests with the documents which is the output stage.