SP500215 Overview of the Second Text REtrieval Conference (TREC-2) chapter D. Harman National Institute of Standards and Technology D. K. Harman Table 1. Document Statistics (disk3) 90,257 78,325 161,021 6,711 Median number of terms per record (diski) 182 353 181 313 82 (disk2) 218 346 167 315 (disk3) 279 358 119 2896 Average number of terms per record (diskl) 329 375 412 1017 89 (disk2) 377 370 394 1073 (disk3) 337 379 263 3543 <smry> Summary: Document will identi[OCRerr] a type of natural language pro- cessing technology which is being developed or mar- keted in the U.S. <narr> Narrative: A relevant document will identify a company or institu- tion developing or marketing a natural language pro- cessing technology, identify the technology, and identify one or more features of the company'S product. <con> Concept(s): 1. natural language processing 2. translation, language, dictionary, font 3. software applications <fac> Factor(s): <nat> Nationality: U.S. <Ifac> <def> Definition(s): <Itop> Each topic is formatted in die same standard method to allow easier automatic construction of queries. Besides a beginning and an end marker, each topic has a number, a short title, a one-sentence description, and a summary sentence or two that can be used as a surrogate for the full topic (often very similar to the one-sentence description). There is a narrative section which is aimed at providing a complete description of document relevance for the 5 assessors. Each topic also has a concepts section with a list of assorted concepts related to the topic. This section is designed to provide a mini-knowledge base about a topic such as a real searcher might possess. Additionally each topic can have a definitions section andlor a factors section. The definition section has one or two of the defi- nitions critical to a human understanding of the topic. The factors section is included to allow easier automatic query building by listing specific items from the narrative that constrain the documents that are relevant. Two par- ticular factors were used in the ThEC-2 topics: a time factor (current, before a given date, etc.) and a nationality factor (either jiwolving only certain countries or excluding certain countries). While the ThEC topics did not present a problem in scal- ing, the challenge of either automatically constructing a query, or manually constructing a query with little fore- knowledge of its searching capability, was a major chal- lenge for ThEC participants. In addition to filtering the relatively large amount of information provided in the topics into queries, the sometimes narrow definition of relevance as stated in the narrative was ditficult for most systems to handie. 3A The Relevance Judgments The relevance judgments are of critical importance to a test collection. For each topic it is necessary to compile a list of relevant documents; hopefully as comprehensive a list as possible. For the TREC task, three possible