MONO91 Problems of Evaluation chapter Mary Elizabeth Stevens National Bureau of Standards that subject indexes are not the most important source, nor even a major source. Herner found, for example, that only about 16 p ercent of his respondents reported use of indexes and abstracts as primary tools in literature searches. He reports, for the use of tools in becoming aware of current sources of information, 477 of 3832 responses indicating the use of indexing and abstracting publications as against 486 using footnotes or other cited references, 1/ 291 using library acquisition lists, and 212 using separate bibliographies (Herner, 1958 [265]). These data, and similar findings of [OCRerr]ishendon that 17 percent of scientists queried considered the scanning of titles in accession lists and announcement bulletins a principal means to find information of interest, 2/ suggest that KWIC type indexes may be adequate for many purposes. On the other hand, the KWIC index to the U.S. Government Research Reports made available to the public on an experimental basis through the Office ot Technical Services was discontinued after a year of subsidized operation because too few of the users indicated willingness to pay a fee in order to have the service continued on a subscription basis. The evaluational problem here involves the lack of information on indexing costs, the relatively few quantitative and objectively validated studies that have been made of user needs, the question of whether what the user says he does or wants is what he really wants or does, and the matter of defining `1interest1 for different users with differing purposes and requirements. The concept of I1interest1.! is taken to mean the motivations of a particular user or group of users at a particular time, while the equally imprecise notion of `1relevance1 refers to the value judgments made by the user as to the relation of an item to his query or interest. A final core problem, then, is that of the question of relevancy itself, involving recognition that tirelevancy is a comparative rather than a qualitative concept . . . (and) ... that a doc[OCRerr]mment of little relevancy in the eyes of X might well be highly relevant in the eyes of Y. [OCRerr]/ Mooers states, similarly, that: "There is no absolute `Relevance' of a document. It depends upon the person and his background, the work and the date. What is not relevant today may be 4/ relevant tomorrow. - Good discusses various possible measures of `relevance' - logical measures, frequency measures, references to, citations of, interest measures, linguistic measures _ 1/ 2/ 3/ 4/ 5/ Note that Herner's data and those of Glass and Norwood, 1958 [232], reporting 6.9 percent use of cross-citations in another paper as the method of learning of important work as against 1.2 percent using an indexing service, appear to re-enforce the claims of those who advocate citation indexing. Fishenden, 1958[197], p. 163. Bar-Hillel, 1959 [33], p. 4-8.4. Mooers, 1963 [423], p. 2. Good, 1958 [234], pp. 7-9. 147