SP500215 GE in TREC-2: Results of a Boolean Approximation Method for Routing and Retrieval chapter P. Jacobs National Institute of Standards and Technology D. K. Harman is probably a case where there is no point in having a Boolean query, and probably systems with the best train- ing and weighting methods do best. Another important point about the TREC-2 results is that the best automatic systems did significantly better than the best manual systems in the routing task, prob- ably because of the success of the training methods used. This raises an important question with respect to the rel- ative contributions of time spent on query construction versus time spent assembling training data. The volume of training data used in TREC-2, with hundreds of thou- sands of relevance judgements, is not realistic for most routing scenarios. Thus, while we should continue to rely oil good training methods, we should be careful to sep- arate out the effects of training and to develop manual routing methods that work with smaller amounts of train- ing data. Given that Boolean and manual methods seem to do best on certain topics, and other approaches that em- phasize weighting do better on others, it makes sense to combine the best from different approaches. ilowever, this raises the issue of how different results can be incor- porated into our model without losing the advantages of the Boolean method, particularly the compatibility with so many existing systems. 5 Future Goals Many 6f our results from TREC-2 suggest areas where the method of generating Boolean queries, especially using corpus data, can be substantially improved. There is a great deal of room for future progress, so we believe that this approach will continue to be viable with respect to other routing and retrieval methods. The main area for research is in continuing to ex- plore new corpus analysis methods. Our corpus analyzer weights single terms. But the Boolean queries depend on combinations of terms, not only in the case of phrases but also to control the effect of ambiguous words. In the con- text of a given query, a single term can often be roughly comparable to the Boolean AND of two or more other terms. Up to this point, we have not quite been able to automate the process of discovering these relationships in the corpus. This is important for both routing and ad hoc retrieval, but especially for routing. Both the routing and ad hoc systems can benefit from the use of new ranking methods, and possibly from ex- ploring hybrid approaches that take advantage of the Boolean method on topics that are well suited to Boolean expression and degrade more gracefully to traditional weighting methods on other topics. In general, the com- bination of methods is something that merits new exper- iments. The routing system could benefit from new training methods. Because the Cornell system did especially well using a training method that produced large numbers of terms and used both positive and negative information, it is possible that this general approach could help our system as well. The ad hoc system suffers mostly from difficulties in handling the topic descriptions; our method of deriving Boolean expressions from the topic descriptions is still ex- tremely crude, and there are many topics for which the approach produced almost useless queries. The perfor- mance of the automatic ad hoc system was even more erratic than that of the manual routing system, but it is very likely that many of the problems can be solved with a lot more work on processing the topic descriptions. Al- though we are loath to direct research at issues that are particular to the formulation of the TREC topics, this work may be necessary to determine the real power of automatically generating Boolean queries. Finally, we are interested in exploring many ways that our corpus analysis and query generation component can be combined with other systems. Because we have fo- cused our attention on query content rather than ranking or retrieval models, we believe that our results could quite likely be used within many other retrieval systems. It is natural to look for such synergy. We have had some pre- liminary collaboration with the UMass team to try to use our queries within the INQUERY system [2], but we still have a long way to go. We will continue to explore such collaborative efforts and to concentrate our own efforts on corpus analysis and building queries. 6 Summary GE's participation in TREC involved the implementa- tion of a number of strategies for creating Boolean queries from the topic descriptions. A statistical corpus analyzer helped to refine queries for both the routing task, and to generate them automatically for the ad hoc task. The simple Boolean retrieval engine performed well, especially in routing. As before, there is tremendous variation in the topic-by-topic results, suggesting that a great deal more research is needed to find how to get the best re- sults in different routing and retrieval scenarios. We are encouraged by the progress of our system, as well as of the overall field, in these experiments, and are hopeful that in the coming years we will learn how to combine our promising results in Boolean approximation and cor- pus analysis with the more mature ranking and retrieval models of some of the other systems. 198