SP500207 The QA System chapter J. Driscoll J. Lautenschlager M. Zhao National Institute of Standards and Technology Donna K. Harman Top ranked evaluation Rus number: UCFQA1 (category B) Num[OCRerr]queries: 19 Total number of documents over all queries Retrieved: 3800 Relevant: 2056 Rel_ret: 517 Recall - Precision Averages: at 0.00 0.5146 atOlO 0.2645 at 0.20 0.1735 at0.30 0.1006 at 0.40 0.0693 at 0.50 0.0107 at 0.60 0.0069 at 0.70 0.0000 at 0.80 0.0000 at 0.90 0.0000 at 1.00 0.0000 Average precision for all points 11-pt Avg: 0.1037 Average precision for 3 intermediate points (0.20, 0.50, 0.80) 3-pt Avg: 0.0614 Recall: at Sdoes: 0.0162 at lSdocs: 0.0362 at 30does: 0.0692 at 100 does: 0.2156 at 200 does: 0.2899 Precision: at Sdocs: 02737 at 15 does: 02211 at 30 does: 02088 at 100 does: 0.1932 at 200 does: 0.1361 Figure 5. Results for Experiment Completed by September Deadline. Top ranked evaluation Run number: 1 Num[OCRerr]queries: 6 Total number of documents over all queries Retrieved: 1200 Relevant: 863 Rel ret: 147 Recall - Precision Averages: at 0.00 0.3486 at 0.10 0.1676 at 020 0.0749 at 0[OCRerr]0 0.0279 at 0.40 0.0000 at 0.50 0.0000 at 0.60 0.0000 at 0.70 0.0000 at 0.80 0.0000 at 0.90 0.0000 at 1.00 0.0000 Average precision for all points 11-pt Avg: 0.0563 Average precision for 3 intermediate points (0.20, 0.50, 0.80) 3-pt Avg: 0.0250 Recall: Exact: 0.1997 at Sdoes: 0.0033 at lOdoes: 0.0062 at 30does: 0.0382 atl00does: 0.1212 at 200 does: 0.1997 Precision: Exact: 0.1225 at Sdocs: 0.1000 at lOdoes: 0.0833 at 30does: 0.1556 atlOOdoes: 0.1550 at 200 does: 0.1225 Figure 6. Results for Six Poorly Performing Queries using Experiment Completed by September Deadline. 205 Top ranked evaluation Run number: 1 Num[OCRerr]queries: 6 Total number of documents over all queries Retrieved: 1200 Relevant: 863 Rd ret: 152 Recall - Precision Averages: at 0.00 0.4585 atOlO 0.2113 at 0.20 0.0775 at 0.30 0.0315 at OAO 0.0000 at 0.50 0.0000 at 0.60 0.0000 at 0.70 0.0000 at 0.80 0.0000 at 0.90 0.0000 at 1.00 0.0000 Average precision for all points 11-pt Avg: 0.0708 Average precision for 3 intermediate points (0.20, 0.50, 0.80) 3-pt Avg: 0.0258 Recall: Exact: 02109 at S does: 0.0072 at 10 does: 0.0128 at 30 does: 0.0469 atlOOdoes: 0.1374 at 200 does: 02109 Precision: Exact: 0.1267 at Sdoes: 02000 at 10 does: 0.1833 at 30does: 0.1944 at 100 does: 0.1683 at 200 does: 0.1267 Figure 7. Results of Keywording with Document Divisions. Top ranked evaluation Run number: 1 Num[OCRerr]queries: 6 Total number of documents over all queries Retrieved: 1200 Relevant: 863 Rel[OCRerr]ret: 157 Recall - Precision Averages: at 0.00 0.40% atOlO 0.2089 at 020 0.0832 at 0.30 0.0340 at 0.40 0.0000 at 0.50 0.0000 at 0.60 0.0000 at 0.70 0.0000 at 0.80 0.0000 at 0.90 0.0000 at 1.00 0.0000 Average precision for all points 11-pt Avg: 0.0669 Average precision for 3 intermediate points (0.20, 0.50, 0.80) 3-pt Avg: 0.0277 Recall: Exact: 0.2167 at Sdoes: 0.0121 at lOdoes: 0.0167 at 30does: 0.0462 atl00does: 0.1392 at 200 does: 0.2167 Precision: Exact: 0.1308 at Sdoes: 0.2667 at lOdoes: 0.1833 at 30does: 0.1833 at 100 does: 0.1650 at 200 does: 0.1308 Figure & Results of Keywording with Paragraph Divisions.