SP500207 Query Improvement in INformation Retrieval Using Genetic Algorithms - A Report on the Experiments of the TREC Project chapter J. Yang R. Korfhage E. Rasmussen National Institute of Standards and Technology Donna K. Harman (2) User involvement and relevance judgment Since our system uses relevance feedback (on the training queries and the ad hoc queries), users (evaluators) are needed to evaluate the retrieved documents. The evaluators are our researchers in the [OCRerr]EC project. The document evaluation process runs like this. On each iteration, for each query individual the retrieved documents are printed with both their full text (or abstracts or summaries if available) and document numbers. An evaluator examines the retrieved documents based on his/her reading of the text (or abstract or summaries) and understanding of the topic, and marks the document numbers which are judged relevant to the topic. The results of the judgment from the evaluators are fed back to the system. The system then applies the genetic modification process (as described in Section 3) based on the feedback information. (3) The feedback process The feedback process can be described as follows. The system uses the relevance judgment to calculate the performance (P) of each query individual based on the function described in Section 3.2. However, the parameters in the formula were changed as follows. First, there are no known relevant documents for each query, the value C is deleted. Second, in our initial tries on a few training topics we found that for these very large databases many nonrelevant documents are retrieved in the first generation. This would cause the individuals which retrieved few relevant and many nonrelevant documents to get lower P values than those which retrieved no documents or only a few nonrelevant documents. So instead of using equal weight for the values of A and B, we gave more weight to A. The formula became: P = 1O*A - B. This increases the chance of survival for those individuals which retrieve a few relevant documents. The weight 10 was chosen for the parameter A because in several initial tries (weight equal to 1, 5 and 10), 10 brought more relevant documents in the modified generation than the other weights. However, values higher than 10 were not used to avoid the query variants to converging too quickly so that the final query might be a local optimum. After computing the performance for each variant, genetic operations are applied to generate a set of new query individuals. Those new query individuals are then used by the system to retrieve documents. However, those documents which have been retrieved in previous generations are not presented in the new generation, even if they satisfy the selection condition. This strategy is similar to the residual collection method used by Chang et. al. (1971). The feedback process continues until either no more relevant documents are retrieved for any query variants or the user (the evaluator) decides to stop the process, mainly due to the time constraints. 38