SP500215 Automatic Routing and Ad-hoc Retrieval Using SMART: TREC 2 chapter C. Buckley J. Allan G. Salton National Institute of Standards and Technology D. K. Harman Run Best > median <median crnlV2 1 38 11 crnlL2 4 40 6 crnlV2-b 9 36 5 Table 1: Comparative Ad-hoc results Global Similarity The crnlV2 run demonstrates the quality of results obtainable with simple methods. The weighting for terms is chosen based upon re- sults from TREC 1. Query terms are weighted by the formula in equation (1) ("ltc" in Smart's vocabulary). Document terms are weighted using a normalized logarithmic term frequency ("lnc" ): d[OCRerr]k = logf[OCRerr]k+1.0 [OCRerr][OCRerr]3i.[OCRerr]1(logfjj +1.0)2 (3) where dik is the weight of term Tk in document D[OCRerr], fik is the occurrence frequency of term Tk in document D[OCRerr], and t is the total number of terms in the collection. The denominator provides normalization of vector length. Note the absence of the "idf" factor log(N/nk). Table 2 shows the results of that weight- ing scheme in crnlV2-b. Regrettably, be- cause of an oversight during the official run (a misnamed inverted file), the official submit- ted run crjilV2 did not use the weighting ap- proach described above (and recommended in our TREC 1 report). Instead crnlV2 acciden- tally used the "idf" factor in both the query and the document. That mistake caused 10% loss in retrieval effectiveness (from a recall- precision average of 0.3512 to 0.3163). Global and Local Cornell's TREC 1 ad-hoc submission in- creased the similarity measure of a query and document if some sentence in the query matched some sentence in the document suf- ficiently well.[1] The result was that any query/document pair which contained a sen- tence match was retrieved before all that did not have such a match. For TREC 2, we hoped to find a less restrictive balance between the global and local similarities. At the same time, mg parts other than sentences, and to investi- gate combining mul[OCRerr]iple local similarities. Our approach is similar to that used in [4]. We built a training collection using the 50 queries from Q2 and the 74,520 documents from the Wall Street Journal included in D2. For each of the 3.7 million query/document pairs, we calculate the global similarity and some set of local similarity values. The least squares polynomials (LSP) approach devel- oped for [4] are used to find the "ideal" c[OCRerr] efficients for the global and local values in the equation: sim = global + [OCRerr] locali + ,@[OCRerr] local2 + ... (The LSP functions actually yield a constant factor which we ignore since it does not affect ranking.) We consider local values from the following broad classes: * Comparing sentences of the query against sentences of the document. In general, we use a simple "tfx idf" weight without nor- malization, though we experimented with other weights. * Comparing paragraphs of the query against paragraphs of the document. (For the most part, each section of the query topic is a separate paragraph.) In this case, we use the weighting of equation 1 above for the query paragraphs, and try a variety of weights for the document para- graphs. * Comparing the query against paragraphs of the document. We use the same weighting schemes as above. We also tried combinations of the above cate- gories: e.g., the best matching paragraph pair and the best matching sentence pair. See Ta- ble 3 for a complete list of local values that were considered. In all, we tried 72 combinations of local and global values, using from one local value to 19 different local values.1 The LSP-determined a and [OCRerr] `5 of those values are then applied to a retrieval run on that same set of queries and documents. The top performing result in- cludes only a single local value: the best match 1There were rougMy 1.2 million possible combina- tions; we chose 72 that seemed, based on earlier exper- we wished to investigate local similarities us- iments, likely to succeed. 48