SP500207 Combining Evidence from Multiple Searches chapter E. Fox M. Koushik J. Shaw R. Modlin D. Rao National Institute of Standards and Technology Donna K. Harman Table 3: Matrix of weighting options Combinations Term frequency Doc. weights Doc. vector norm. nnn tf none none ntn tf tfidf none npn tf prob none bnn binary none none btn binary idf none bpn binary prob none mnn max[OCRerr]orm none none mtn max[OCRerr]orm idf none mpn max[OCRerr]orm prob none ann augnorm none none atn augnorm idf none apn augnorm prob none 3.3 CPU time The retrieval runs required approximately 4 minutes of CPU time for each topic. We did a full sequential pass through the document vector file for this since we did not have enough disk space for the inverted file. 3.4 Combination of Run Results Our original plan was to compare several schemes for combining the results of a number of runs. The results of using one scheme, the CEO Model, are reported elsewhere in these proceedings by Paul Thompson. One of our goals was to use an artificial intelligence methodology called Decision Trees. Decision Trees should reflect the most effective evaluation methods for each query. Our Decision Trees were produced by a commercial software package called KnowledgeSEEKER [2], by FirstMark Technologies Limited. Input to KnowledgeSEEKER is a training set of documents for which the results of the various evaluation runs are known along with the relevance values of the documents. The output of KnowledgeSEEKER is a Decision Tree that indicates in what order to apply the evaluation runs in order to determine the relevance of documents in the collection. The highest level of the tree is the test that most effectively evaluates documents for a given query. Based on the results of the most effective test, documents may be assigned a relevance score or additional tests that further evaluate the documents may be suggested. The Decision Tree partitions the collection into disjoint document groups with relevance values for the documents in each group. The results produced by KnowledgeSEEKER for the tralning set of documents must be parsed in order to apply the Decision Trees to other documents in the collection. In Phase 1 training data was fed into the Decision Tree system so that the ranges of values of independent variables (e.g., similarity for a particular type of search) could be categorized into sets or intervals that predict the dependent variable values (e.g., relevance value of 0 or 1). For example, one Decision Tree developed during Phase 1 is given below. 322