SP500207 LSI meets TREC: A Status Report chapter S. Dumais National Institute of Standards and Technology Donna K. Harman 3.3.1 Separate vs. combined scaling We used 9 separate subscalings for the TREC experiments. Our decision to use the 9 subcollections was largely a pragmatic one initially. Would like to create a single large scaling and compare the results with those obtained using the subcollections. 3.3.2 Centroid query vs. many separate points of interest A single vector was used to represent each query. In some cases the vector was the average of terms in the topic statement, and in other cases the vector was the average of previously identified relevant documents. A single query vector can be inappropriate if interests are multifacted and these facets are not near each other in the LSI space. In the case of the routing queries, for example, we could match new documents against each of the previously identified relevant documents separately rather than against their average. We have developed techniques that allow us to match using a controllable compromise between averaged and separate vectors (Kane-Esrig et al., 1991). We did not use this method for TREC, but would like to do so for TREC-2. 3.3.3 Interactive inteifaces All LSI evaluations were conducted using a non-interactive system in essentially batch mode. It is well known that one can have the same underlying retrieval and matching engine, but achieve very different retrieval success using different interfaces. We would like to examine the performance of real users with interactive interfaces. A number of interface features could be used to help users make faster (and perhaps more accurate) relevance judgements, or to help them explicitly reformulate queries. (See Dumais and Schmitt, 1991, for some preliminary results on query reformulation and relevance feedback.) Another interesting possibility involves returning something richer than a rank-ordered list of documents to users. For example, a clustering and graphical display of the top-k documents might be quite useful. We have done some preliminary experiments using clustered return sets, and would like to extend them to the TREC-2 collection. The general idea is to provide people with useful interactive tools that let them make good use of their knowledge and skills, rather than attempting to build all the smarts into the database representation or matching components. 4. Onward to TREC-2 We were quite pleased that we were able to use many of the existing LSIISVD tools on the TREC collection. The most important finding in this regard was that the large, sparse SYD problems could be computed without numerical or convergence problems. The computations were not fast, but a day of CPU on existing workstations (for each subcollection) is certainly feasible, especially given that these calculations are only done once at the beginning to create the database. We have already computed some larger SYDs (100k x 200k) and would like to take advantage of this for TREC-2. In terms of accuracy, LSI performance was reasonable. There are some obvious ways to improve 149