ISR11 Design Criteria for Automatic Information Systems chapter M. E. Lesk G. Salton Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. V-3 user needs and reactions by users to initial search efforts cannot usefully be taken into account in order to improve the service. The SMART document retrieval system which has been operating on an IBM 7O9[OCRerr] for the last two years has been used extensively to test a large variety of automatic retrieval procedures, including fully automatic information analysis methods, automatic procedures for dictionary construc- tion, and iterative search techniques based on user interaction with the system.[3,[OCRerr],5,6] The evaluation results indicate that presently held assumptions concerning the design of information systems are untenable, and point the way to alternative design criteria. Some of the experiments conducted with the SMART system are outlined briefly, and the principal results are described in the remainder of this study. 2. The SMART Experiments SMART is a fully automatic document retrieval system operating on the IBM 7O9[OCRerr][OCRerr]. The system does not rely on manually assigned keywords or index terms for the identification of documents and search requests, nor does it use primarily the frequency of occurrence of certain words or phrases included in the document texts. Instead, the system goes beyond simple word-matching procedures by using a variety of intellectual aids in the form of synonym dictionaries, hierarchical arrangements of subject identifiers, statistical and syntactic phrase generating methods, and the like, in order to obtain the content identifications useful for the retrieval process. Stored documents an& search requests are then processed without any prior manual analysis by one of several hundred automatic content analysis methods, and those documents which most nearly match a given search request