CRANV2 Methods for presentation of results chapter Cyril Cleverdon Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. -35 of situations can be taken from these tests. Relevance decisions were based on four levels of relevance; ff we consider Relevant 1 documents, there are 12 such documents relevant to the 42 questions of subset 2. Relevance 1 and 2 documents come to 57, Relevance 1, 2 and 3 documents total 154 and Relevance 1-4 documents come to 198. It can be "seen that changing the decision as to the relevant documents (a + c) materially alters the proportion of relevant documents in the complete collection (N). On the other hand, the collection size can be changed. Originally there were 1400 documents in the collection. A subset of the collection was formed which consisted of 200 documents; a characteristic of this subset was that it retained all of the 198 documents that were relevant to the 42 questions of subset 2. While the number of relevant documents is now held constant, the proportion changes because of the reduction in the document collection from 1400 to 200 documents, it is convenient to express this variation as a parameter, and this is the aforementioned Generality lmmber i.e. 1000 (a + c) , the total relevant documents divided by the collection N size, with a constant. This parameter is not a measure of retrieval performance, but one which reflects the environment of the relevance decisions made; e.g. if the generality number for a set of questions is 5, this means that there are, for each question, an average of five relevant documents for every thousand documents in the collection, irrespective of what the actual size of the collection might be. For the example given above, the change from the larger to the smaller collection size (bearing in mind that there are 42 questions} changes the generality 1000 x 198 1000 x 198 number from 42 x 1400 - 3.4 to 42 x 4-4-90"[OCRerr]=c[OCRerr] 23.6. Therefore, as far as retrieval performance is concerned, the significance of a change in either the relevance decisions or the collection size is that in both cases it is the generality number which alters. The single performance measures that can be used can be listed as follows:- a a+c usually known as Recall Ratio; at Western Reserve University it is called 'Sensitivity'. and has also been called 'Hit Rate'. 12 a ÷ C complementary to recall ratio. Called by Fairthorne, 'Snobbery Ratio', a a+b now generally known as Precision Ratio, formerly called by Cranfield 'Relevance Ratio', "Also described as 'Pertinency Factor' or 'Acceptance Rate'.