ISR11 The SMART System -- Retrieval Results and Future Plans chapter G. Salton Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 1-6 to consider the context within which these results were obtained be[OCRerr]ore their general validity is accepted. It is,in fact, possible to argue that the results are completely invalid because in many cases no real user need existed when the requests were formulated; because the searches were conducted in an artificial environment rather than within an operational system; because the collection sizes used were in all cases very small, consisting of less than 1000 documents for each collection; because the dictionaries used to perform the word normalization were in some cases not constructed independently of the collections; because some of the relevance judgments used to compute recall and precision may be suspect since they were not always generated by actual users of the system; because the'original manual indexing available for the aerodynamics collection may not have been performed under ideal conditions; and because in a situation in which it is impossible to alter one gi[OCRerr]ven variable without also affecting many others, it is difficult to make positive statements whose general validity is unchallengeable. In fact, the situation is not nearly so complicated as these objections appear to indicate. Most of the searches in fact exhibited a quite consistent behavior over a l[OCRerr]ge range of experiments involving many changes of variables. Thus concept or synonym dictionaries were constructed for three subject fields in several diiferent ways, and dictionaries constructed from one sample collection were used on a different new collection with substantially similar results : syno[OCRerr]'m recognition was always found to be superior to raw word stem matches. Relevance judgments, evaluating the usefulness of documents with