CRANV1P1 Comments chapter Cyril Cleverdon Jack Mills Michael Keen Cranfield An investigation supported by a grant to Aslib by the National Science Foundation. Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. - 115 - variations hopefully evened out by having nearly two hundred different questioners In an investigation that had very similar objectives, Salton went to the other extreme in his original tests. Using only seventeen questions in the general field of the test collection, these questions were specially prepared for the test and did not represent any actual requirements. The set of 400 documents in the collection were then assessed against these prepared questions by a number of students, this assessment being based only on short abstracts. Since the searching was also done on the abstracts, there was obviously the probability of even more distortion than was the case with the source document questions of Cranfield I. The interesting point, however, is that this seemingly crude technique of question preparation and relevance assessment did, in fact, allow a considerable amount of useful data to be obtained concerning the performance characteristics of the various index language options, and this data appears to be sufficiently valid for certain conclusions to be reached. When this evidence is added to that obtained from Cranfield I, there are some grounds for suggesting the possibility that everyone is over-emphasising the importance of relevance assessments in experimental testing, and that, however relatively unscientific the method used, reliable information can be obtained. It is intended to investigate this point in future work at Cranfield by having various people make new relevance assessments of the document-question sets used in the present project. The search results can then be re-scored on the basis of these new - and presumably somewhat different - relevance assessments, and analysis will show whether the comparative performance of various index languages is thereby affected. In experimental testing, the common practice, not unnaturally, is for the groups to work with document collections with which they have some familiarity, and this project was no exception. The language of aerodynamics might be said to fall some- what to the left of centre in regard to its precision, it is, in fact, mushy rather than firm. A[OCRerr] such it presented a number of difficulties; not only could one find the same notion being expressed in different, ways by different authors, one often had the situation where the same notion was expressed in different ways in the same paper. Discussing this point with one of thq authors, he said that certain people considered it good style if, after expressing a notion in the title in one way, a new phrase could be used for the abstract and another phrase be found for the actual text. Even without this particular complication, the subject matter was full of semantic problems. An illustration of this is provided by a question (not in any way a-typical) which, as originally received, read 'Has anyone investigated relaxation effects on gaseous heat transfer to a suddenly heated wall' ' When asked to suggest alternative search terms, the questioner sen{ back the following comments. Relaxation effects. Could be replaced by 'excitation of internal molecular energy modes (or states)'. The excitation could result from collisions between gas molecules alone, or gas molecules and molecules in the solid. Gaseous heat transfer. 'Gaseous' could be omitted, but does help to limit the field. 'Hear'could be replaced by 'energy', 'transfer' by 'conduction' or 'transmission'. Suddenly heated wall. 'Suddenly' could be replaced by 'rapidly', 'heated' by 'cooled' and 'wall' by 'solid'