SP500207 Retrieval Experiments with a Large Collection using PIRCS chapter K. Kwok L. Papadopoulos K. Kwan National Institute of Standards and Technology Donna K. Harman document-focused QTD learning of c(AB) based on known relevant queries in q(A). The ad hoc queries q(B), the total dictionary, and the total collection c(AB) then form a network using 1Cm weighting, from which both automatic and manual ad hoc query retrieval results (W[OCRerr]auto and W1[OCRerr][OCRerr]) are obtained. (c) Feedback Queries: To simulate user relevance feedback, we employ the ten highest ranked sub- documents of the automatic ad hoc results in (b), and determine their relevance to their respective topics manually. Two queries out of 25 have no relevant documents within the first ten, and they are removed leaving 23. We have performed two types of feedback training: without and with query expansion. In both cases, query-focused and document-focused training were done. Our method of adding terms to existing queries is based on [8,9] during DTQ direction training. The n[OCRerr] documents relevant to a query q, are instantiated to I and their activation to each term node, after gated through the weights wk[OCRerr]=d[OCRerr]kIL[OCRerr], is averaged. The highest 20 activated terms with document frequency less than 2000 are then used to expand qa. However, not all the terms are different from the existing terms of qa, so that on average, each query got expanded with about 12.3 terms (284 new terms for 23 queries). After this, training proceeds in the QTD direction. We have not used expansion for documents because usually there is only one relevant query per document. We have done only one round of feedback iteration. 3. System Description Our system was designed over a number of years for IR experimentation. Its primary goal is to be flexible so that unforseen developments or algorithms can be supported for testing without much change in the basic implementation. Many intermediate files are produced for more convenient hooks to and from the system. These will be trimmed in later verisons. It was originally programmed in Pascal, but has been translated and is now running in a UNIX and C environnient on a Sun SparcStation 2GS. The system has 48MB RAM, a TREC-dedicated 1.7GB disk drive, plus about 0.5GB on another drive that we can occasionally use. Our system characteristics and timings are described in the Appendix of this paper. 4. Results and Discussions Our runs are named PIRCS'n', where n=l denotes fully automatic retrieval, n=2 denotes manual, i.e. automatic combined with soft-boolean where boolean queries are manually created, n=3 denotes feedback using automatic retrieval, and n=4 is the same as n=3 but with query expansion. Although we use sub- documents for indexing and retrieval, our final 200-document output list has only unique document ID's; all sub-documents of the same document ID are suppressed except for the one with the highest rank. Precision-recall and other measures are gathered in the Appendix of this volume. Our runs are based on twenty five queries for ad hoc and another twenty five for routing retrievals, which we recognize as too few and may not reflect sufficiently the variety of query types in real-life. On the other hand, the evaluation curves and average values serve as a lower bound to the power of our system because relevance judgments are rendered only to the combined first 100 documents of all systems. This means documents ranked 101 to 200 in our retrieval are automatically considered irrelevant if they are not in the judged relevant set. Also, Category B texts come from a single source, viz. WSJ of multiple years, not like Category A with texts from multiple sources. With these reservations we like to make the following observations: (a) Inter-site comparison shows that the PIRCS results are the best among Category B participants for both ad hoc and routing retrievals, and comparable to the best methods in Category A based on one evaluation with crnlB. 160