SP500215 Overview of the Second Text REtrieval Conference (TREC-2) chapter D. Harman National Institute of Standards and Technology D. K. Harman Overview of the Second Text REtrieval Conference (TREC-2) Donna Harman National Institute of Standards and Technology Gaithersburg, MD. 20899 1. Introduction InNovember of 1992 the first Text REtrieval Conference [OCRerr]EC-1) was held at NIST [Harman 1993]. The confer- ence, co-sponsored by ARPA and NIST, brought together information retrieval researchers to discuss their system results on a new large test collection (the TIPSTER col- lection). This was the first time that such groups had ever compared results on the same data using the same evalua- tion methods and represented a breakthrough in cross- system evaluation in information retrieval. It was also the first time that most of these groups had used such a large test collection and therefore required a major effort by all groups to scale up their retrieval techniques. The overall goal of the TREC initiative is to encourage re- search in information retrieval using large-scale test col- lections. It is hoped that by providing a very large test collection, and encouraging interaction with other groups in a friendly evaluation forum, new momentum in infor- mation retrieval will be generated. Because of the NIST involvement, groups with commercial retrieval products have participated in TREC, leaning to increased techno- logical transfer between the research labs and the com- mercial products. TREC has also provided a state-of-the- art showcase of retrieval methods for ARPA clients. Whereas the TREC-1 conference demonstrated a wide range of different approaches to the retrieval of text from large document collections, the results should be viewed as very preliminary. Not only were the deadlines for re- sults very tight, but the huge increase in the size of the document collection required significant system rebuild- ing by most groups. Much of this work was a system en- gineering task: finding reasonable data structures to use, getting indexing routines to be efficient enough to index all the data, finding enough storage to handle the large in- verted files and other structures, etc. Still, the results showed that the systems did the task well, and that auto- matic construction of queries from the topics did as well as, or better than, manual construction of queries. The second TREC conference [OCRerr]EC-2) occurred in Au- gust of 1993, less than 10 months after the first confer- ence. In addition to 22 of the TREC-l groups, nine new 1 groups took part. bringing the total number of participat- ing groups to 31. Many of the original TREC-1 groups were able to "complete" their system rebuilding and tun- ing, and in general the TREC-2 results show significant improvements over the TREC-l results. This paper provides an overview of the TREC-2 conf&- ence, including a review of the TREC task, a brief de- scription of the test collection being used, and an overview of the results. The papers from the individual groups should be referred to for more details on specific system approaches. 2. The TREC Task 2.1 Int[OCRerr]duction TREC is designed to encourage research in information retrieval using large data collections. Two types of re- trieval are being examined -- retrieval using an "adhoc" query such as a researcher might use in a library environ- ment, and retrieval using a "routing" query such as a pro- file to filter some inconling document stream. The TREC task is not tied to any given application, and is not primar- ily concerned with interfaces or optimmed response time for searching. However it is helpfiil to have some poten- tial user in mind when designing or testing a retrieval sys- tem. `The model for a user m TREC is a dedicated searcher, not a novice searcher, and the model for the ap- plication is one needing monitoring of data streams for in- formation on specific topics (routing), and the ability to do adlioc searches on archived data for new topics. It should be assumed that the users need the ability to do both high precision and high recall searches, and are will- ing to look at many documents and repeatedly modify queries in order to get high recall. Obviously they would like a system that makes this as easy as possible, but this ease should be reflected in TREC as added intelligence in the system rather than as special interfaces. Since TREC has been designed to evaluate system perfor- mance both in a routing (filtering or profiling) mode, and in an atihoc mode, both functions need to be tested.