SP500207 Natural Language Processing in Large-Scale Text Retrieval Tasks chapter T. Strzalkowski National Institute of Standards and Technology Donna K. Harman Run nyuirl [OCRerr] nyuir2 } nyuir3 Name brute__[OCRerr] prune j_uniform Queries 25 [OCRerr] 25 J 25 Tot number of docs over all queries Ret 4986 4990 4989 Rel 3318 3318 3318 ReiRet 1161 958 1039 Recall Precision_Averages 0.00 0.7830 0.7168 0.7368 0.10 0.6216 0.4858 0.5592 0.20 0.4891 0.3722 0.4660 0.30 0.3815 0.2032 0.3235 OAO 0.2350 0.1382 0.2099 0.50 0.1871 0.0599 0.1385 0.60 0.1447 0.0419 0.0801 0.70 0.0705 0.0058 0.0560 0.80 0.0229 0.0030 0.0056 0.90 0.0000 0.0000 0.0000 1.00 0.0000 0.0000 0.0000 Average precision for all points 11-pt 0.2669 0.1843 0.2341 Average precision at 0.20,0.50,0.80 3-pt 0.2330 0.1450 0.2034 _________ Recall at 5 docs 0.0713 0.0444 0.0571 l5docs 0.1326 0.0929 0.1297 30 docs 0.1868 0.1424 0.1734 lOodocs 0.3350 0.2550 0.2918 200 docs 0.4294 0.3466 0.3908 Precision at S docs 0.5680 0.5200 0.5360 15 docs 0.5200 0A267 0.4907 30 does 0.4320 0.3627 0.4147 100 docs 0.3140 0.2684 0.2916 200docs 0.2322 0.1916 0.2078 Table 5. Ad-hoc run statistics with Automatic Brute Force Merge, Uniform Merge with hand pruning, and Automatic Uniform Merge. 185 Run nyuirl {nyui14[OCRerr] nyuirs Name brute concepts negations Queries 25 j251 25 Tot number of docs over all queries Ret 4986 4984 4984 Rel 3318 3318 3318 RelRet 1161 1291 1309 Recall Precision Averages 0.00 0.7830 0.7685 0.7823 0.10 0.6216 0.6521 0.6625 0.20 0.4891 0.5396 0.5460 0.30 0.3815 0.4306 0.4419 OAO 0.2350 0.2671 0.2755 0.50 0.1871 0.2094 0.2085 0.60 0.1447 0.1457 0.1457 0.70 0.0705 0.0660 0.0660 0.80 0.0229 0.0385 0.0385 0.90 0.0000 0.0000 0.0000 1.00 0.0000 0.0000 0.0000 Average precision for all points 11-pt [[OCRerr] 0.2669 0.2834 0.2879 Average precision at 0.20, 0.50,0.80 3-pt [OCRerr] 0.2330 0.2625 0.2643 Recall at S docs 0.0713 0.0738 0.0741 l5docs 0.1326 0.1362 0.1386 30docs 0.1868 0.2007 0.2011 100 does 0.3350 0.3513 0.3565 200 does 0.4294 0.4739 0.4828 Precision at 5 docs 0.5680 0.6080 0.6080 l5docs 0.5200 0.5360 0.5493 30 docs 0.4320 0.4760 0.4773 100 does 0.3140 0.3432 0.3484 200 does 0.2322 0.2582 [OCRerr]__0.2618 Table 6. Ad-hoc run statistics using Automatic Brute Force Merge: without Concepts field, with Concepts field, and with Concepts excluding negated terms.