TREC-8 Interactive Track Searcher Questionnaires
Here are html versions of the minimum data collection instruments the Rutgers team has
prepared for use by all participants in the TREC-8 Interactive Track.
Please note that these data collection instruments are the
MINIMUM required of all participants. All sites are free to
add to these instruments, and to add any others they would
like. We suggest, however, that all additions be made to
the END of each of these questionnaires, and that new instruments
be applied AFTER these, as appropriate.
Please note that the SEARCHER WORKSHEET is the method that we
have specified for people to note the instances as they discover
them. This is a requirement of the track (we want to know what
people think instances are, and to compare them to what the
evaluators think they are), but it doesn't need to be met
by this form if, for instance, a system offers the opportunity
for such labels on-screen.
- The instruments are to be administered as follows:
- Pre-Search Questionnaire _before_ any searching has been done
on either system (e.g. immediately after your informed consent
form, or the verbal fluency test).
- Post-Search Questionnaire immediately _after_ each individual
search on each system (there will be eight for each subject)
- Post-System Questionnaire _after_ the final (i.e. fourth)
search on each system, and _after_ the post-search questionnaire
for that search (there will be two for each subject)
- The Tutorial Worksheet and Experimenter Note Form are optional;
if used, there will be one Tutorial Worksheet for each system
tutorial, and one Experimenter Note Form for each subject.
- Searcher Worksheet will be given to each subject at the same
time that each subject is presented with each topic. Since
some systems will surely be asking searchers to indicate
"instances" on-line, we do not anticipate that all sites will
use this form as is. We suggest that in such cases, the instruction
in this form be given to the subjects orally before each search,
modified as relevant to conform to the interface being used.
Depending on the nature of the interface, the instruction could
be included in the space where "instances" are recorded.
- Exit Questionnaire _after_ the second post-system questionnaire
has been administered. In this questionnaire, we have deliberately
used two system names which have no obvious ordering associated
with them. We strongly suggest the use of such "neutral" system
names at all sites.
- Additional questionnaire items specific to each site should be
added to the _end_ of each of these questionnaires, or administered
in separate questionnaires _after_ each of these, as appropriate.
Interviews and other data collection instruments should _follow_
the ones indicated here (e.g., an "exit interview" should _follow_
the "Exit Questionnaire", _not_ come before it).
For information about this webpage contact Paul Over
Last updated: Tuesday, 01-Aug-2000 07:13:19 MDT
Date created: Monday, 31-Jul-00