SP500207 Overview of the First Text REtrieval Conference (TREC-1) chapter D. Harman National Institute of Standards and Technology Donna K. Harman test documents are released and should not be modified after that point. The unmodified queries should be run against the test documents and the results submitted to NIST. 2. Method 2-- manual initial query construction. adhoc queries -- The query is constructed in some manner from the topic, either manually or using machine assistance. The methods used should be identified, along with the human expertise (both domain expertise and computer expertise) needed to construct a query. Once the query has been constructed, it will be sub[OCRerr] mitted to the system (with no manual intervention), and the results from the system will be the results sub[OCRerr] mined to NIST. There should be no manual intervention after initial query construction that would affect the results. [OCRerr]anual intervention is covered by Method 3.) routing queries -- The queries shou[OCRerr]d be constructed in the same manner as the adhoc queries for Method 2, but using the training topics, relevance judgments, and training documents. They should then be submitted to MST before the test documents are released and should not be modified after that point. The unmodified queries should be run against the test documents and the results submitted to NIST. 3. Method 3-- automatic or manual query construction with feedback. adhoc queries -- The initial query can be constructed using either Method 1 or Method 2. The particular technique used should be described. The query is submitted to the system, and a subset of the retrieved documents is used for manual feedback, i.e., a human makes judgments about the relevance of the docu- ments in this subseL These judgments may be communicated to the system, which may automatically modify the query, or the human may simply choose to modify the query himself. In either case, the exper- tise of the person or persons examining the documents should be described, both their domain expertise and their experience in online searching, and the manner of system feedback (i.e., automatic system modification of query or human modification) should be also described. At some point, feedback should end, and the query should be accepted as final. Three sets of results should be sent to NIST for each topic. The first set should be the results without feed- back, i.e., the top 200 documents retrieved from an initial query produced without feedback, whether pro- duced manually or automatically. This set should be exactly the same as the results from Method 1 or Method 2, but should be submitted again as one part of Method 3. The second set should be the results after only one iteration of feedback, with the top X documents used in the first iteration of feedback frozen. For example, if you "used't the top 20 documents for feedback, then the second set of results should have these documents as the top 20 documents, followed by the top 200 documents retrieved based on feedback. The term "used't means all documents for which some information has been seen by the judger, and are deemed by the system to have been seen. These two sets of results will be used by MST to calculate a residual evaluation measure. The third set of results should be a record of your feedback, i.e., a list of documents in the exact order they were seen and judged, with an indication of iteration boundaries. For example, if you ran six iterations of feedback, with 10 documents looked at for each iteration, the record would be a list of the 60 documents seen by the 91user11, marked at 10, 20, 30, etc. You should also indicate what information the user communicated to your system about each document (relevant/not relevant, too general/too specific/on target, etc.). We will be specifying a format for these record files later. These files will be used to calculate measures based on the total number of relevant documents retrieved both across iterations and across a given document level. routing queries -- Method 3 cannot be used for routing queries because routing systems have typically not supported feedback. In general these guidelines served well, although there was some misunderstanding about what constituted feed- back. The guidelines will be clarified for TRBC-2. 4