TREC-8 Interactive Track Searcher Questionnaires

1. Please note that these data collection instruments are the MINIMUM required of all participants. All sites are free to add to these instruments, and to add any others they would like. We suggest, however, that all additions be made to the END of each of these questionnaires, and that new instruments be applied AFTER these, as appropriate.
2. Please note that the SEARCHER WORKSHEET is the method that we have specified for people to note the instances as they discover them. This is a requirement of the track (we want to know what people think instances are, and to compare them to what the evaluators think they are), but it doesn't need to be met by this form if, for instance, a system offers the opportunity for such labels on-screen.
3. The instruments are to be administered as follows:
   • Pre-Search Questionnaire _before_ any searching has been done on either system (e.g. immediately after your informed consent form, or the verbal fluency test).
   • Post-Search Questionnaire immediately _after_ each individual search on each system (there will be eight for each subject)
   • Post-System Questionnaire _after_ the final (i.e. fourth)
   search on each system, and _after_ the post-search questionnaire for that search (there will be two for each subject)
   • The Tutorial Worksheet and Experimenter Note Form are optional; if used, there will be one Tutorial Worksheet for each system tutorial, and one Experimenter Note Form for each subject.
   • Searcher Worksheet will be given to each subject at the same time that each subject is presented with each topic. Since some systems will surely be asking searchers to indicate "instances" on-line, we do not anticipate that all sites will use this form as is. We suggest that in such cases, the instruction in this form be given to the subjects orally before each search, modified as relevant to conform to the interface being used. Depending on the nature of the interface, the instruction could be included in the space where "instances" are recorded.
   • Exit Questionnaire _after_ the second post-system questionnaire has been administered. In this questionnaire, we have deliberately used two system names which have no obvious ordering associated with them. We strongly suggest the use of such "neutral" system names at all sites.
4. Additional questionnaire items specific to each site should be added to the _end_ of each of these questionnaires, or administered in separate questionnaires _after_ each of these, as appropriate. Interviews and other data collection instruments should _follow_ the ones indicated here (e.g., an "exit interview" should _follow_ the "Exit Questionnaire", _not_ come before it).

Nick Belkin

Last updated: Tuesday, 01-Aug-2000 13:13:19 UTC

Date created: Monday, 31-Jul-00