MONO91 Problems of Evaluation chapter Mary Elizabeth Stevens National Bureau of Standards 7. Z. 4 Summary: Potential Advantages as Bases for Evaluation In view of the difficulties engendered by the underlying core problems, the criticisms that can be brought against tests of "retrieval effectiveness", the general lack of comparative data and standards of measurement, the question of evaluation of automatic indexing procedures largely reduces to the weighing of potential advantages and disadvan- tages. In the case of such procedures as KWIC and citation indexing, some of these possibilities, both pro and con, have been discussed previously. In general, suggested bases for evaluation reflecting operational considerations may be summarized as follows: 1. Speed and timeliness Z. Relative economy 3. Consistency and reliability[OCRerr]l/ 4. Elimination of the need for further human intellectual effort after initial planning and programming has been done. 5. Providing a product that could not otherwise be obtained. 6. Ease of updating and revision of indexes so produced.- From the point of view of possible operational advantages, these may be combined into the single criterion: The achievement of a more effective and more economical balance between the meeting of the objectives of the indexing system and the utilization of available resources. 1/ Compare McCormick, 196Z [OCRerr]4O9], p. 18Z: "A computer is objective in its operations and it can be repetitive. If given a certain amount of information about a document, it is always able to index the document in a consistent manner. This consistency is desired so as to avoid the situations where a person might index a document differ- ently on various occasions, or where it would be indexed differently by another person when there appears to be no good reason for a difference." Note, however, O'Connor's point previously mentioned, (1963 [443], p. 16): "It has been argued that mechanized indexing has the advantage of consistency... However this argu- ment by itself says very little in favor of mechanized indexing. For two humanly produced index sets for a document which differ somewhat may both be quite useful, though imperfect, while the index set which the same program will always reproduce for the same document may be worthless." See, for example, Youden, 1963 [658], p. 332: "The facility with which indexes may be updated and the ease of selecting items for special bibliographies will result in the majority of indexes being computer produced before many years." 156