IRE An experiment: search strategy variations in SDI profiles chapter Lynn Evans Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Experiment 287 Documents The documents used were not special in any way and were obtained on a weekly basis from the INSPEC current file. The most important consideration was that the document collection should be typical of the INSPEC database. Eight weeks' documents were taken in two groups of 4 weeks each separated by a time gap. Four consecutive weeks was considered a reasonably adequate span in that the cycle of most journals would thereby be covered. The reason for having two separate 4-weekly groups was to allow for profile performance analysis and modifications in between the first 4 and last 4 weekly SDI runs. In all over the 8 runs more than 20000 documents were matched against the profile file, the individual weekly totals being 3095, 2640, 2582, 2194, 2542, 2781, 2123, and 2860. As with many aspects of this experiment deciding the size of the document collection was empirical. Later, when the pattern of retrieval performance was found to be similar from week to week it was not considered worthwhile completing the performance calculations for all 8 runs so that 20000 documents would seem to have been more than an adequate number and something like 5000 would have been quite acceptable. Since then of course the question of document collection size has been thoroughly considered as part of the design of an `ideal' information retrieval test collection6-7 Queries It was considered important that the profiles should be based on real rather than artificial questions and that they should operate in a near-real' environment. This was because one of the main objectives was to detect differences between the retrieval performances of the search strategies which showed through despite the hazards associated with real user assessments. Experience in the INSPEC SDI investigation had highlighted the fact that real users' relevance assessments were often not entirely based on subject content but might be influenced by such factors as the language of the original document, its source, the author's reputation, etc. It was also not unknown for slight changes of interest not to be notified and only become apparent on investigation of poor profile performance. Despite these known irritations which only tend to cause confusion the feeling was that they are an inevitable feature of operational systems and should not be ignored by the use of artificial queries. On the other hand it was not thought politic to use paying customers of INSPEC's commercial SDI service because of the danger of their alienation when asked to assess documents over and above those retrieved by their optimum profiles. The compromise decided was to recruit a user group from UK university research workers, the arrangement being that for their agreement to provide subject interest statements and to assess the relevance of document outputs they would gain, at no cost, experience of a mechanized current awareness service. In selecting the university research workers the latest issue of the Department of Education and Science's annual publication Scientific Research in British Universities and Colleges was used as a random source of names, addresses and subject interests. Of the 100 people originally approached (40 Physics, 40