IRE An experiment: search strategy variations in SDI profiles chapter Lynn Evans Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Results 301 (averages of numbers) were also calculated. Perhaps it should be pointed out that the precision values were rather superfluous since, with the ranked- output cutoff procedure, all the available information is contained in the recall figure-at a particular cutoff point if the recall figure for one strategy is better than that for another then the precision figure is automatically similarly so. The single performance figure usually associated with the ranked-output cutoff procedure is normalized recall1 5. Strictly this requires knowledge of the positions of all relevant items in the ranked output. The data used here were the recall figures at 9 cutoff points down to the 55th ranked document. Taking the average of the recall figures at the nine cutoff points gave an `average' recall value which approximated to the normalized recall in its effect of ranking the search strategies in an order of merit. This order together with the approximated normalized recall values for the eight non-boolean search strategies for runs 1, 5 and 6 are shown in Table 14.3. No great weight can be given to the actual values of the approximated normalized recall since they depend on the number and positions of the cutoff points used. However with different cutoff points the relative positions of the search strategies would not be expected to change. TABLE 14.3. Ranking of search strategies by normalized recall [OCRerr]ased on 9 cutoff poInts, averages of Dumbers) Order Relevance RI documents Relevance RJ/2 documents of merit Search strategy (normalized recall) Search strategy (normalized recall) Runi RunS Run6 Runi RunS Run6 1 TWC (51.5) TWC (58.5) GWC (56.4) TWC (44.5) GWC (49.7) TWC (47.8) 2 GWC (51.4) GWC (57.3) TWC (54.7) GWC (44.2) TWC (49.2) GWC (47.5) 3 GTWC (49.0) GTWC (55.7) GTWC (53.4) COW (42.5) COW (46.9) CTW (45.9) 4 CRTW (47.1) CRTW (53.0) COW (52.0) CTW (42.4) CTW (46.3) COW (45.8) 5 COW (47.0) CTW (52.8) CO (50.3) CRTW (41.9) CRTW (45.6) CO (44.3) 6 CTW (46.1) COW (52.6) CTW (50.0) CO (40.1) OTWC (45.2) CRTW (43.9) 7 CO (43.1) CO (48.6) CRTW (49.4) CT (39.4) CO (43.8) CT (43.7) 8 CT (41.8) CT (47.3) CT (48.5) OTWC (39.4) CT (42.6) OTWC (43.2) I I Some points to emerge from Table 14.3 are: (1) The two strategies TWC (term-weight cumulation) and GWC [OCRerr]roup- weight cumulation) are always the best, occupying first and second positions on all three runs for both relevance Rl and Rl/2 documents. (2) Strategy CT (simple co[OCRerr]ordination of terms) performs the worst, always being in one of the last two positions. (3) Perhaps surprisingly, strategy CRTW (co[OCRerr]rdinate matching of restricted list of terms) holds its own with the others, mostly taking the middle positions. (4) Strategy GTWC k%roup/term `powers of 2' weight cumulation) is unusual * in being always third best for relevance Ri documents but well down the lists when considering R1/2 documents. This suggests that the harsher consequences of `powers of 2' weighting are not particularly suited to retrieval of `in-between' partly relevant documents.