IRE Retrieval system tests 1958-1978 chapter Karen Sparck Jones Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Ir The decade 1958-1968 221 projects are considered in detail in the next chapter, and so will be referred to here only as far as is necessary to give a coherent picture of the work done. The tests carried out by Schuller7, the Syntol group8, Altmann9, Lancaster12' 13, and Shaw and Rothman11 are significant because of their character or consequences. Tests like Blagden's10, Spencer's37 and Newell and Goffman's38 can be regarded as representative minor ones of the period. In terms of the concerns of the period, the most important of these tests were those with the objective of comparing indexing languages, typically some or all of more conventional classifications, or pre-coordinate subject headings, with newer types of classification and post-coordinate indexing, particularly using a thesaurus. Thus the focus of Schuller's test was a comparison between the UDC and Uniterms, of Cranfield 1 a comparison between the UDC, alphabetical subject headings, a facetted classification, and Uniterms; the Cranfield-WRU test6, which can be labelled Cranfield 1+, compared WRU telegraphic abstracts (essentially representing role-link indexing) with facets, Cranfield 2 a whole range of languages falling into broad groups based on natural language or controlled language terms, Altmann the `ABC' pre-coordinate system with a simple KWIC index, Shaw and Rothman pre- and post-coordinate natural language and also a simple KWIC index, and CWRU telegraphic abstracts with manual keywords, automatic keywords, a `meta-language' or controlled language, and subject headings. Spencer compared conventional abstract journal classification schemes with SCI. In the role/link subgroup Montague's test19 also involved comparisons with other types of indexing. The other tests in this subgroup, by Herner et al.1 7, Sinnett1 6, Cohen et al.1 8, and van Oot et al.20 involved various combinations of terms, links, and roles. From one point of view these could be described as different indexing languages, but compared with the much larger differences of language studied in the other tests, the link/role tests could be deemed studies of a single type of language. More straightforward evaluations of single languages were those of relational indexing by the Syntol group, simple post-coordinate terms by Blagden, and the Medlars controlled language by Lancaster. The motivation for the comparative tests tended to be to demonstrate the superiority or at any rate competitiveness of the more novel approaches involved, for example the use of facets in classification, of a post-coordinate thesaurus as opposed to pre-coordinate subject headings, of telegraphic abstracts, or of the SCI. Non-comparative tests like Blagden's were intended to show that a new method could provide satisfactory performance. In most cases the tests were concerned with effectiveness, but in some cases costs were explicitly investigated. Projects interested in the convenience or competitiveness of novel approaches were often implicit cost evaluations. In general, the assumption made in these tests was the one mentioned earlier: that the index language is a major, or perhaps the crucial, factor influencing performance, so that the choice of language is very important. A further closely related assumption was that indexing languages need to be sophisticated, though the contrary was occasionally held, for example by Shaw and Rothman. The specific substantive assumption made by the different projects then tended to be that the particular type of sophisticated language advocated was desirable, e.g. one using links and roles, or the more sophisticated relations of Syntol. Only the more extensive comparative tests