SP500215 Retrieval of Partial Documents chapter A. Moffat R. Sacks-Davis R. Wilkinson J. Zobel National Institute of Standards and Technology D. K. Harman results to see if it was helpful to use the rank of the whole documents along with the rank of its component parts. Experiment 1: Rank full documents against the queries using standard cosine measure. Experiment 2: Split documents into sections. Measure similarity of each section against the queries using standard cosine mea- sure. Order documents based on the highest ranked section. Experiment 3: Split documents into sections. Measure similarity of each section against the queries using standard cosine measure. Order documents based on 0.5[OCRerr][OCRerr]1wt(s,,) where S,, is the flth highest ranked section in the document. The effect of this formula is that a document's weight is determined by a decay formula using the weight of all of a document's sections. (Values other than 0.5 were tried, but gave poorer performance.) Experiment 4: Split documents into sections. Measure similarity of each section against the queries using standard cosine measure. Then weight each section using its type, for example (introduction) or (address). Order documents based on tw(i[OCRerr]pe(s,, ))0.5"[OCRerr]1wt(s,,) where S,, is the flth highest ranked section in the document, type(s,,) is the type of the sec- tion, and tw(t[OCRerr]) is the weight of the type t[OCRerr] The weights of the types of sections were ob- tained by conducting a set of experiments where all types but one were given a weight of 1, and in turn, each type was given a weight of 2. Using these experiments it was determined that (purpose) and (summary) were each more helpful, and that (misc) was less helpful. As a result, in this experi- inent tw((purpose)) = tw((summary)) = 2, tw((misc)) = 0.5, and other weights were set to 1. Experiment 5: Rank the documents, and rank the sections. Form a new rank based on the average rank of these two ranks. 189 Other experiments were carried out using best two sections, and formulas that more closely ap- proximated the cosine measure. None of these experiments achieved better results than the ones displayed here. The obvious conclusion is that if documents are available for ranking as whole doc- uments, then for this collection it is preferable to do so. 3.3 Section retrieval For very long documents it may be desirable to return relevant sections rather than relevant doc- uments. We were interested to see whether it might be useful to know about the rank of the containing document. In the first experiment documents were ranked, and sections shown in document order. This produced very poor re- sults. Next, we still ranked sections in higher ranked documents ahead of lower ranked docu- ments, but used section ranking for sections in the same document. This was reasonable but there were still many irrelevant sections being ex- amined. Finally, we attempted to delete these ir- relevant sections by using document ranking, and then section ranking, but this time discarding sec- tions that had a section rank of greater than 200. Experiment 6: Rank sections against the queries using standard cosine measure. Experiment 7: Rank full documents against the queries using standard cosine measure. Order the sections by their appearance within documents. Experiment 8: Rank full documents against the queries using standard cosine measure. Order sections, first by document, then by rank within documents. Experiment 9: As in experiment 3, but then delete all but the 200 highest ranked sec- tions. These experiments show that ranking both documents and sections does help to find more relevant sections. This result is in contrast to the earlier investigation of finding relevant docu- ments. We are able to find relevant sections much easier if the rank of both the sections, and the containing documents are taken into account. 3.4 Paged versus section retrieval Our results showing that retrieving documents based on section ranking was not as useful as