SP500207 Natural Language Processing in Large-Scale Text Retrieval Tasks chapter T. Strzalkowski National Institute of Standards and Technology Donna K. Harman terms from queries while preparing them for a routing run. (2) Topics 51-75 were processed using the same 3 fields as above with respect to the test database only (disk 2). They were then run in ad-hoc mode independently against the training data- base and the test database, and the top 200 documents from each run (for each query) were merged using document scores. The upper 200 documents were selected for the final result. It should be noted that we run topics against disk 1 using idf weights on terms from disk 2 (a roufing-like mode). This, it turned out, created an even distribufion of hits between the two databases. We call this method the uniform merge. (3) We repeated run (2), but this time we run topics 51-75 in ad-hoc mode on both disks (i.e., different weights were used in these runs). Again, top 200 documents were selected. We call this method the bruteft)rce merge. (4) Finally, we repeated uniform merge with queries manually pruned before actual retrieval. Summary statistics for these runs are shown in [OCRerr]tbles 4 and 5. It has to be pointed out that these results are lower than expected because of various problems with the database search prograin and because the Concepts field in TREC topics was not included in search queries. Subsequent runs with Concepts field included in search produced results shown in Table 6. ACKNOWLEDGEMENTS We would like to thank Donna Harinan of NIST for making her IR system available to us. We would also like to thank Ralph Weischedel, Marie Meteer and Heidi Fox of BBN for providing and assisfing in the use of the part of speech tagger. Jose Perez Carballo has contributed a number of valuable observations during the course of this work, and his assistance in processing the TREC data was critical for our being able to finish on time. This paper is based upon work supported by the Defense Advanced Research Project Agency under Contract N00014-90-J-1851 from the Office of Naval Research, under Contract N00600-88-D-3717 from PRC Inc., and the National Science Foundation under Grant IRI-89-02304. We also acknowledge support from Canadian Institute for Robotics and Intelligent Systems (IRIS). 184 Run nyuirl nyuir2 Name routing routing Queries 25 25 Tot number of does over all queries Ret 5000 5000 Rel 3766 3766 RelRet 834 [OCRerr] 1177 Recall Precision Averages 0.00 0.5691 0.6983 0.10 0.3618 0.5072 0.20 0.1846 0.3508 0.30 0.1314 0.2997 0.40 0.1065 0.2286 0.50 0.0481 0.1481 0.60 0.0295 0.0604 0.70 0.0104 0.0296 0.80 0.0104 0.0260 0.90 0.0000 0.0125 1.00 0.0000 0.0000 Average precision for all points 11-pt 0.1320 0.2147 Average precision at 0.20, 0.50, 0.80 3-pt 0.0811 0.1750 Recall at 5 docs 0.0338 0.0374 l5docs 0.0671 0.0961 30docs 0.0979 0.1533 lOodocs 0.2117 0.3119 200docs 0.2988 0.4298 Precision at Sdocs 0.3360 0.4480 15 docs 0.2960 0.4587 30 docs 0.2787 0.3973 lOodocs 0.2276 0.3080 200docs 0.1668 0.2354 Table 4. Routing run statistics: (1) duplicate terms (unintenfionally) removed from queries; and (2) du- plicate terms retained and Concepts fields included.