SP500207 TIPSTER Panel -- The University of Massachusetts TIPSTER Project chapter W. B. Croft National Institute of Standards and Technology Donna K. Harman * The documents in TIPSTER They collie the general arc[OCRerr]a are the [OCRerr] Tall Street Journal, technology area, Departnicnt are heterogeneous ill terms of both subject and length. of science, technology and economics, but the sources Associated Press newswire, Ziff magazines in the high of Energy abstracts, and the Federal Register. * The queries (known as `topics" in TIP STER) are longer and have more structure than those found in other test collections. * The queries have specific and These criteria (specified in the between relevance judges but information retrieval Svsteiii. strict cutena specified for documents to be relevant. "narrative" part of the topic) will reduce inconsistency are sometimes difficult to handle in the context of an * The routing experinielits are unhke any carried out before. * The retrieval and routilig experiments [OCRerr]vith Japanese are also unique. The first TIPSTER. evaluatioii w('s liujited by a number of factors, the primary one being the lack of relevance judgeiiieiit.[OCRerr] for the jiutial query set. This made it difficult to carry out experiments to sele([OCRerr]t technique.[OCRerr] appropriate for large, full-text databases. The results from this evaluation 5lL()1l1(l[OCRerr] therefore, be regarded as preliminary, and indeed raise more questions than they answer. In the retrieval experinient, 50 new [OCRerr]tol)ics" were used to search the "old" database, which consisted of approxiiiiately 1 GByte of text. One of the major subjects of the eval- uation was to try different forms of queries produced by processing the topics. Our basic approach to topic processing is to parse them, selecting parts to be indexed, recognizing phrases and "factors" such a." locations, dates, companies, etc. Some factors, such as "de- veloping country", which have been specifically identified as important in the topic, will be expanded using a synonym operator. Weights reflecting relative importance are attached to the concepts (words and phrases). Phr[OCRerr]ise-based concepts are represented by operators defined in the inference net l[OCRerr]u1guage. These operators use proximity of the words making up the phrase as the major form of evidence for the presence of the concept [1]. The result of topic processing is an inference net representing the information nee(l. In addition to the automatic query processing, some query versions were generated by simulating simple user interactioii with the results of the topic processing. The modifica- tions to the automatically processed topics were limited to changing the weight of concepts, deleting concepts considered uniniportant, and adding structure (such as specifying syn- onymous concepts). The lilost signifi([OCRerr]ant change in the last category was the introduction of "unordered window" 01) er[OCRerr]i.tors to Si umlate paragraph-level retrieval. The equivalent in terms of a user user interface would 1)e to ask users to group concepts that should occur together. The results of the first evaluation are described here in terins of the average precision in the top 5, 30 and 200 documents in the ranking produced by the inference net retrieval engine (INQUERY). This evaluation method was chosen because only the top 200 documents for each query were judged for relevance. The results were as follows: 102