MONO91 Problems of Evaluation chapter Mary Elizabeth Stevens National Bureau of Standards "Indexing 0£ the remaining cases in the experiment will be performed by machine from full text, using the Type I list of discard words and the Type II list to pre- pare an analysis of the frequencies related to index-word space. Instead of selecting specific words as indexing terms, concepts will be selected (statistically) as volumes in index-word space. A rough physical analogy to this process would be to toss pennies at the previously mentioned grid so that, for every Type II word in the source document, a penny lands at its proper slot on the grid. Where the pennies heap up in a pile, you have a concept.,' "Searching will be carried out essentially by indexing a question presented narratively, determining the concept volumes that represent the question, and searching those volumes in document space for the relevant document numbers. Since the `edges' of the concept volumes are determined statistically, output can be listed in order of probable relevance; as an option the question could be accompanied by a request that `at least 100 references be supplied', in which the concept boundaries would be adjusted to provide that number." 1/ case It will thus be noted that the proposed indexing and search program begins on a derivative basis to establish for one-half the experimental material the significant words, next combines word frequency with significant word distance data to derive probabilistic association factors between words, then develops clusters, and finally indexes the items in terms of the clusters rather than words so as to provide assignment rather than extraction of index terms. 7. PROBLEMS OF EVALUATION We have noted, in the introduction to this report, that several fundamental and highly controversial questions can be raised with respect to the feasibility and evaluation of any automatic indexing scheme and with respect to the evaluation of any indexing systems whatsoever. Y[OCRerr]t if automatic indexing procedures are to be based upon previous [OCRerr]iuman indexing or if their results are to be compared with human results, then the questions of the quality, the reliability and the consistency of human indexing are crucial ones indeed. Thus, Solomonoff warns: "The finding of exact languages for retrieval is also made less likely, in view of the fact that the categorizations of documents that are presented to the machine as a training sequence will not be performed altogether consistently by the human cataloger." 2/ Montgomery and Swanson ask whether human indexers are in fact self-consistent and consistent with each other, and they suggest: 1/ 2/ Eldridge and Dennis, 1963 [182], pp. 97-99. Solomonoff, [OCRerr]959 [[OCRerr]62], pp. 9-TO. 143