CRANV1P1 General Considerations chapter Cyril Cleverdon Jack Mills Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. (which correspond to the devices being investigated in this project) - are used. The question of exactly what constitutes a different system will therefore be discussed later. Considered separately, certain conclusions drawn from Cranfield I may be difficult to justify, since it is possible for different interpretations to be placed on the evidence. Consider the matter of the relatively equal performance that could be obtained by the four systems. It has been argued, quite reasonably, that the un- naiLiral relationship between the questions and their related documents was such that there was little difficulty in locating the source document by whichever method it was indexed, and that this was the reason for the level performance. The advantage of an experimental situation is that, in a well-designed test, it is reasonably simple for different hypotheses to be tested, and, by analysis of the search failures, it was simple to show that recall (which was the main objective in Cranfield I) is far more dependent on the concept indexing than on the index language. Therefore. since the concept indexing was in general the same for all four systems, the first step had been taken to ensure that the performance should be much the same for all four systems. It is not intended to re-argue the conclusions listed above. It is sufficient to say here that they appeared reasonable as a basis for future work. All hinged on the twin factors of recall and precision. Why this should be the case has aroused a considerable amount of argument, and many different suggestions have been made regarding the criteria that are of importance in the evaluation of an information retrieval system. Bourne (ref. 7) presented a long list of such possible criteria and asked, "It is not clear why so much attention has been given to recall and rele- vancy. Should these be regarded as better criteria than any of the others proposed?" We would suggest that all criteria fall into one of two groups. The first group, which we call user criteria, is made up of those factors which are of concern to the users of a system. Such criteria are related to the operational performance of the system and can be listed as follows:- 1. The ability of the system to present all relevant documents (i.e. recall) 2. The ability of the system to withhold non-relevant documents (i.e. precision) 3. The interval between the demand being made and the answer being given (i.e. time) 4. The physical form of the output (i.e. presentation) 5. The effort, intellectual or physical, demanded of the user (i.e. effort). The second groupie made up of criteria in which the ordinary user is not direct- ly interested and which are therefore the sole concern of the managers of the sys- tem, that is to say all those who decide the policy, finance the system, or are in any way responsible for or participate in the actual operation of the system. The user is not normally concerned with the intellectual methods that are adopted to achieve a particular result, nor is he interested in the economics of the techniques used. Such matters are, however, of major concern to the management, but, on the other hand, they cannot be considered in isolation or as an end in themselves. It is a reasonable assumption that an I.R. system basically exists for the purpose of meeting the requirements of the user group, and any evaluation of management criteria must always be made in relation to the effect which they have on the user criteria. It cannot, for instance , be argued that one indexer is better than another without relating their indexing to the requirements of the users of the system.