MONO91 Problems of Evaluation chapter Mary Elizabeth Stevens National Bureau of Standards be en to study an existing system (e.g., using Merck, Sharp and Dohme data) with respect to indexing terms such as "penicillin," "toxicity," and "mode 0£ action." He then attempts to define various possible machine assignment rules, and then to determine the probable over-and-under assignments that would result from the application of these rules. Typical results pertinent to both questions of word-indexing evaluation and of inter- indexer consistency showed that for 23 documents indexed under the term "toxicity," 11 did not contain the stem "toxi. . ." at all; that 17 items indexed under "penicillin" contained the word at least once; that none of 34 randomly selected documents not indexed under "penicillin" contained the word, but that 7 of 28 items not so indexed but selected as probable candidates from title and other clues did contain the word. (O'Connor, 1961 [447]) Typical suggestions, comments, and conclusions made by O'Connor include the following.' "It might be required that the mechanized indexing permit as good (or no worse) retrieval as existing human indexing, because it is desired to free the subject- skilled indexing personnel for other work. Or poorer retrieval (than possible with human indexing such as is presently done of comparable material) might be accepted from computer indexing, because poorer retrieval is better than none 11 and there is a shortage of subject-skilled people to do the additional indexing." - "Such considerations as the following are relevant. Over-assigning can increase input costs and storage (to an extent dependent on the storage system), but mechanizing indexing might be worth the cost. Over-assigning might also increase the number of irrelevant documents retrieved, but the increase might be insignificant," 2/ Suppose terms A, B, and C each correctly characterize five percent of a ten thousand document collection, each term is overassigned to another five percent, and over-assignment of each term occurs independently of the correct assigning and over[OCRerr]assigning of the others. Then about nine documents will be extra for the search question A & B & C." 3/ "The question of permitting some under-assigning, that is, the computer failing to assign E a term] T to some document which should have it, is more delicate. Human indexers sometimes underassign. If we knew the rate of ounderassigning by human indexers for a term T, we might consider allowing the computer a similar rate. However, some cases of underassigning might be more important than others and if the computer made more important mistakes than the human indexers, retrieval might not be `good enough'." 4/ 1/ 2/ 3/ 4/ O'Connor, 1960E444], p.3. O'Connor, 1961 [448], p. 199. O'Connor, 1960 [444], p. 6. Ibid, pp. 6-7. l[OCRerr]2