SP500207 Workshop on: Automatically Generating Adhoc and Routing Queries report of discussion group Susan T. Dumais National Institute of Standards and Technology Donna K. Harman Workshop on: Automatically Generating Adhoc and Routifig Queries [Summary by: Susan T. Dumais, Belicore, std@bellcore.comJ About 20 people attended the two workshops on automatic query generation. Many different issues were addressed, and I've tried to organize the important points under a few general headings. Topic Statements: We spent some time initially talking about how the topics statements were developed, what retrieval scenarios they are representative of, and some consequences of this for research. The topic statements are much more detailed, structured, and specific than queries associated with most previous IR test collections, averaging about 150 words in length. Most topics (routing topics 001-025 and adhoc topics 051-100) require that fairly specific facts be retrieved. Routing topics 026-050 are more general. The topic statements were generated by subject domain experts and reformulated using search results from two different retrieval systems. While this might be characteristic of routing applications or of dedicated searchers, there was some question about how likely more casual users would be to generate such queries. There was some interest in developing a companion set of shorter topic descriptions that could be used to better explore the effects of term expansion, feedback, and iterative query formulation. In contrast, there was also some interest in having expert human searchers carry out much deeper searches for a few topics in order to cast a wider net and increase the variety of documents retrieved. Term Extraction: Most of the fields in the topic description were used, and there was some evidence that the <concept> field was the most useful. Almost all systems used a stop-list and some kind of stemmer. A few systems recognized and tagged common abbreviations or acronyms, proper names, company names, place names, etc. Everyone agreed that a compendium of this information would be a valuable common resource. Many systems used differential term weighting. Typically weights derived from a statistical analyses of the documents were also used to weight query terms. Term weights sometimes depended on the topic field or syntactic slot the term occupied. About half of the systems used phrases in addition to single words. Phrases were usually derived by simple statistical means using word adjacency (or co-occurrence withing k positions), with high thresholds on overall frequency of occurrence to limit the number of phrases. Some systems used syntactic analysis to discover phrases, but most of these groups did not automatically generate their queries. Phrases appeared to improve performance somewhat by increasing both precision and (somewhat unexpectedly) recall. Term expansion: Term expansion has long been used to increase recall by making the search query more comprehensive. Not all relations are equally useful in expansion, and the most commonly used relation was synonymy. Queries were expanded using several different sources of information - a thesaurus to generate semantic categories; a general, manually-constructed lexical system (wordnet); associations automatically derived from an analysis of word usage in the documents or 367