SP500215 TREC-2 Routing and Ad-Hoc Retrieval Evaluation using the INQUERY System chapter W. Croft J. Callan J. Broglio National Institute of Standards and Technology D. K. Harman of the INQUERY system and it indicates that the automatic query processing has improved considerably. The combination search (1NQ042) is slightly worse than INQO41 at the 5 document cutoff level, but overall is better than either the automatic or manual queries on their own. An adhoc search that incorporates automatic paragraph-level matching was also tested in TIPSTER and this resulted in a further 5% improvement. INQ023 and 1NQ024 are routing query sets that were created automatically using rel- evance judgements from volumes 1 and 2. In addition to the single-word terms added in INQOO3, 10 phrase-level concepts and 20 paragraph-level concepts were added to the query. A phrase-level concept is a #UW5 two-word pattern that occurs frequently in the relevant documents, and a paragraph-level concept is a [OCRerr]UW50 two-word pattern. The #UWn operator looks for co-occurrence in any order in a text window of size n. The difference between INQ023 and 1NQ024 is that 1NQ023 contains the original query terms in addition to terms extracted from relevant documents, whereas 1NQ024 contains only terms from relevant documents. Query Type 1NQ023 1NQ024 5 Docs .67 .68 (+1.5%) Average Precision 30 Docs 100 Docs 11-Pt Avg .60 .47 .38 .59 (-1.7%) .46 (-2.2%) .39 (+2.6%) Table 4: Results for TIP STER routing queries These results show that there is little difference between using the original query or just the relevant documents. This is probably due to the large number of relevance judgements available in this routing experiment. In a relevance feedback situation, where there are far fewer relevant documents, the original query is very important. It is clear that the addition of phrase and paragraph-level structure to the routing has improved performance. The average precision for 1NQ023 is 8.6% higher than INQOO3. Combining these new runs with manually modified routing queries produced further improvements. 6 Summary The TREC-2 runs, both in the adhoc and routing categories, provided further evidence that manually generated queries are not, in general, superior to automatically processed natural language queries. In the case of routing, in fact, the manual queries are significantly less effective. They do, however, improve the effectiveness of retrieval when used in combination with the automatic queries. This combination of query types has been a theme of the research at the University of Massachusetts and has been established as effective in a number of experiments. The additional TIPSTER runs showed that learning structure in the form of phrases and paragraph-level co-occurrences is effective for routing. They also showed that learning techniques significantly improve performance (the best routing runs were more than 20% 82