ISR11 Design Criteria for Automatic Information Systems chapter M. E. Lesk G. Salton Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. v-b with the SMART system: a) IRE - 1 : a set of about 4oo abstracts of documents in the computer literature published in 1959, used with approximately 20 search requests; b) IRE - 2 a set of about 400 abstracts of documents in the computer literature published in 1960 and 1961, used with approximately 20 search requests; c) ADI : a set of 82 short papers in documentation, each approximately 2000 words long, presented at the 1963 Annual Meeting of the American Documentation Institute, and processed against 35 search requests; d) Cranfield - 1 : a set of 200 abstracts of documents in aero- nautical engineering previously used by the Aslib-Cranfield project L12], and processed against [OCRerr]2 search requests; e) Cranfield - 2 : a set of 1200 additional document abstracts in aeronautical engineering, similar to the abstracts included in the preceding collection. It is seen that these collections fall into three distinct subject areas: computer science, documentation, and aeronautical engineering. The ADI collection in documentation is of particular interest because full papers are available rather than only document abstracts. The Cranfield collections, on the other hand, are the only ones which are also manually indexed by subject experts, thus permitting a comparison of the standard keyword search procedures with the automatic text processing methods. The evaluation results obtained with the first four of these collec- tions are summarized in the next section.