CRANV1P1 Additional Tests chapter Cyril Cleverdon Jack Mills Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 110 - to indicate that it ,:.i,,.i [OCRerr]hi:: particular reference. However, when a document number appears ir[OCRerr] .:-.[OCRerr];ec[OCRerr]o[OCRerr] with another cited reference, the number is not repeated, but a tick ..:: i:[OCRerr] [OCRerr]n the appropriate column. For instance, in Fig. 7.3, it can be seen i!: ,.; i[OCRerr]-als document has two references in common with document 1163, 1164, 1639 c.[OCRerr]m 171[OCRerr]i, three references in common with document 2379, and four references in ,::ommon with document 17!5. To return to Fig. 7.2, when document 1067 had been ¢[OCRerr]ntered, then this number was crossed off and the reference card was re-so[OCRerr]ed in the pack under the next number, namely 1077; again this number was crossed off when the master card had been entered for document 1077, and the reference card re-filed on the next number and so oa until all the document numbers had been entered. The final stage was zo go through the master cards and prepare the bibliographic coupling card (Fig. 7.4). This showed the master document and all the other documents with which it had two or more references in common. It is clearly a matter for argument as to how a citation index should be tested operationally, but within the context of these experimental investigations, it was relatively simple to decide on the method to be used. Our concern was how a citation index operated in regard to recall and precision and the procedure adopted was as follows. For a certain question, the relevant documents were known as well as their relevance level. The numbers of the relevant documents were written across the score sheet as shown in Fig. 7.5, referring to question 34. In order to avoid complexity, a fairly simple example has been taken, where there were six documents all of relevance 3. The numbers in the left hand column indicate the coupling strengrth going from a maximum of 6+ down to i, which latter represents citation indexing. The appropriate bibliographic coupling cards were then taken from the pack, the first of these relating to document 1067. As can be seen in Fig. 7.4, document 1715 had a match of 4 with document 1067;.there were no other documents at this level of match, and since document 1715 is also a relevant document to question 34 (see Fig. 7.5), this is counted as a success and the score is entered appropriately. The document which matches at a level of 3 is not relevant, so this now makes the score one relevant and one non-relevant. At a match of 2, three of the documents are relevant (1164, 1639 and 1716), so the score here becomes four reievant and two non-relevant. By referring to the cards shown in Fig. 7.3, we can calculate the number of documents involved with a single match. There are no olher relevant documents in this set, but many non- relevant, so the score for this is shown aa four relevant and thir[OCRerr]:y-two non- relevant. This process is repeated for all the other relevant documents, as shown in Fig. 7.5. When this has been done, the scores can be totalled to give a set of figures where obviously the maximum recall and the lowest precision will be obtained at a match of 1, and maximum precision with lowest recall obtained at a match of 6+. ttowever, there are various approaches that can be taken in compiling the score, and these wi]l be considered in the volume of test results. Such analysis was done for documents of .all degrees of relevance. In bibliographic coupling as discussed by Kessler, account is only taken of the actual match rather than what might be called the proportional match. For instance, twp review-type articles may each have fifty references, as against two other papers which have only three references. If the former pair of papers have five references tn common, this would be considered a stronger coupling stren[OCRerr]h than the latter pair