IRS13 Test Environment chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 1-4 responsible for the 42 requests in use with the Cran-l collection. The document collection consists not of the authors' own research papers, but of a number of the earlier papers that the author cited in the bibliography to his paper. The first set of seventeen computer science requests were prepared for the iRE-l collection by three project staff members. Two persons made up thirteen of the requests without any knowledge of how the system would per- form in practice1 but having extensive knowledge of the technique of operation of the system. Requests were devised to cover a cross-section of the major topics in the collection but were not "source document" requests in the sense that they were based on particular documents. A third person made up four of the requests using sets of classification headings that had been manually assigned to the IRE-l collection. A second set of seventeen requests was prepared by one person hired for the task. This person was one of two persons who also prepared the 35 docu- mentation requests. Requests were again not based on any one document in the collection. No guidance concerning the length of the requests was given - the hired persons tended to devise requests that were rarely longer than a sentence or two, whereas requests prepared by[OCRerr] staff members were often longer. Techniques used for obtaining relevance decisions are described in part 3. A much more detailed analysis of the characteristics of documents and requests is possible. Such an investigation is given in Section x, part 3, for the ADI documentation requests only. Direct performance comparisons between collections have not formed the purpose