IRE An experiment: search strategy variations in SDI profiles chapter Lynn Evans Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 28[OCRerr] An experiment: se[OCRerr]rch strd'tegy vd'riations in SDI profiles Flectrical/Flectronics, 20 Computers/Control) the 55 who actually supplic(l statements of their subject interests comprised 23 from Physics Departments, 27 from Electrical/Electronics Departments, and 5 from Computer/Control Departments. Given the inevitable delay between their original undertaking t&) participate in the experiment and their receipt of the first notifications for assessment, the response from users was quite gratifying. Over the 8 runs relevance assessments were received from, on average, 82 per cent of participants, the returns for the individual runs being 84, 89, 86, 87, 76, 78, 78 and 76 per cent. As with the number of documents the decision to use about 50 users was `gut-feeling' rather than statistically reasoned. The question of what is an acceptable number of queries cannot of course be considered in isolation and is intrinsically bound up with the number of documents against which the queries are matched. If Q represents the number of queries and D the number of documents it would be tempting to speculate that there is an optimum value for the product QD with minimum acceptable values for Q and D. However it is not only a matter of the numbers of queries and documents but also how many documents are relevant to particular queries. The fewer relevant documents there are the less confidence there can be in the particular recall or precision ratio so there is great doubt associated with, say, a recall figure of 1/1 which could so easily have been 0/1. These are matters which can only be completely controlled in an artificial situation and the statistical bases of relevance assessment have been considered in detail more recently8. Search strategies The strategies to be compared should ideally represent different degrees of intellectual effort when compiling profiles and, also, should require different degrees of sophistication of computer facility. The strategies selected for comparison were all variations of two basic types, viz. those consisting of a single list of terms, and those containing groups of terms, where the groups represent subject concepts in the original query. The list of search strategy types was: (1) Co-ordinate matching of terms without weights (CT) The output is ranked in order of term co-ordination level, i.e. in order of the number of matching profile and document terms. (2) Term-weight cumulation (TWC) The profile terms are assigned weights in accordance with their relative importance to the query. The weights of all matching terms are `summed' to produce a `document score'. The output is ranked in order of document scores. (3) Co-ordinate matching of terms with weights (CTW) The profile terms are weighted as in (2). The output is ranked, first in order of term co-ordination level (i.e. number of matching terms), and then by sum of all matching-term weights.