CRANV1P1 Test Design chapter Cyril Cleverdon Jack Mills Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 14 3. A prepared question, that is a question which has been composed specifically for the purpose of the test and is not a question which meets an actual need of the ques- tioner. Such prepared questions may or may not be based on a particular document or documents. Method of Relevance Assessment I By the questioner II By the consensus of opinion of a group of people III By an individual, not the questioner IV By matching the indexing with the search programme. Type of Individuals) Involved A User of a system B Scientific or technical staff, not users of the system C Librarians or other information staff. If we now chart Type of Question against Method of Relevance Assessment, the various possibilities can be shown Method of Relevance Assessment In the chart, the upper half of each box represents the type of person asking the ques- tion, the lower half represents the type of person making the relevance assessment. An additional variable concerns the type of document on which the reference decision is based, for this can be either a The complete text ft An abstract y The title. It can be seen that the Documentation Inc. example discussed above was, presum- ably, the use of an actual question (1A) where the relevance assessment was made by an individual, not the questioner (III) who was a member of the information staff (C), probably basing his decisions on document titles,making up the code (lA)(IIICy). For Cranfield I the code would have been (3B)(IBa), which is to say that prepared questions were used (3), based on complete documents (a), this resulted in the rele- vance being determined by the questioner (I) and the individuals involved were tech- nical staff not concerned with the system (B).