SP500215 TREC-2 Document Retrieval Experiments using PIRCS chapter K. Kwok L. Grunfeld National Institute of Standards and Technology D. K. Harman NewAdhoc - Revised Routing pii[OCRerr]s5 Pii[OCRerr][OCRerr] % pii[OCRerr]7 % pirc[OCRerr] Total number of documents over all 50 queries Retrieved: 50000 50000 50000 50000 Relevant: 10489 10489 10489 10785 Rel_ret: 7098 7476 +5-3 7385 + 4.0 8279 Interpolated recall - precisron averages: 0.0 .765 .814 + 6A .793 +3.7 .823 0.1 .554 .628 +13.3 .594 + 7.2 .561 0.2 A91 .546 +11.2 .534 +8.6 .505 0.3 .421 .469 +11.4 .469 +11.4 .456 OA .380 A13 + 8.7 .416 +9.5 .417 0.5 .336 .363 + 8.0 .368 +9.5 .368 0.6 .270 .310 +14.8 .314 +16.3 .320 0.7 .212 .240 +13.2 .241 +13.7 .246 0.8 .150 .165 +10.0 .171 +14.0 .160 0.9 .079 .099 +25.3 .098 +24.0 .087 1.0 .014 .006 -57.1 .015 + 7.1 .012 Average precision (non-interpolated) over all rel does .318 .355 +11.6 .350 +10.1 .344 Precision at S does: .600 .660 +10.0 .624 +4.0 .612 10 .582 .632 + 8.6 .598 + 2.7 .572 15 .545 .617 +13.2 .572 + 5.0 .573 20 .527 .583 +10.6 .563 + 6.8 .564 30 .507 .553 + 9.1 .534 +5.3 .540 100 .402 .439 +92 .427 +62 .468 200 .334 .360 +7.8 .353 + 5.7 .3% 500 .222 .238 +7.2 .232 +4.5 .264 1000" .142 .150 + 5.6 .148 +4.2 .166 R-precislon Exact .358 .385 + .385 + .378 Table 2: Revised Routing and New Ad Hoc Retrieval Results and 53.7% of the operational maximum. The R-precision Exact calculates the precision value at x retrieved. where x is the known number of relevants for each query, and can be compared with the theoretical value of 1.0. The `Upperbound' retrieval pircs7 (suggested by Sparck- Jones) means perfonning learning from the known Dis1:3 (not Disk 1&2) relevant documents before retrieval. In other words, we assume the answer documents are known for training and represent the best that probabilistic they can provide using our system. This is however not the true upperbound [Spar79] for retrieval from Disk3. because the vocabulary and usage statistics are still those of Disk 1&2. The vocabulary is retained for comparison with routing 240 results. pircs7 retrieval achieves average precision of 0.350, which improves over pircss (training from Disk 1&2) by about 10% in average precision and about 4% in relevants retrievei of corse in real life. the answer documents are not known; but it is interesting to note that query expansion using Disk 1&2 documents can provide similar performance, showing the imp&tance of query expansion. We later concentrate on routing and discover that additional gains can be achieved by fine tnning of the parameters in our model. For learning: 1) we find that our original method of using only `nonbreak' documents in the given set of relevants actually outprfonns other doccnent selection strategies including using all relevants, `break six' or