IRE Information Retrieval Experiment The pragmatics of information retrieval experimentation chapter Jean M. Tague Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Decision 9: How to analyse the data? 91 .i[OCRerr][OCRerr]igned to recall-- 1. Because of the erratic nature of low recall values for iI[OCRerr]i[OCRerr] small sample of 4 queries, the latter course is chosen. Similarly, the .[OCRerr]vcrdge of the precision values for repeating recall values are used, obtaining ` precision of 0.656 for recall=0.692 and 0.398 for recall=0.846. Linear iliterpolation values [OCRerr]* for the standard recall values r* are obtained from [OCRerr])t)5crved recall and precision points by using the following formula: r* -r1 l'[OCRerr] Pi+ r2 - r1 (P2-Pi) where r1 and r2 are the recall values immediately to the left and to the right ()fr* andp1 andp2 the corresponding precision values. `Pessimistic' precision v[OCRerr][OCRerr]1ues are those associated with the recorded recall value immediately ((Ilowing (i.e. greater than) the standard one. The values obtained by the two iiiethods are shown in Table 5.4. l[OCRerr]ABLE 5.4 .[OCRerr]iondard recall Linear precision Pessimistic precision 1.0 0.75 I 0.892 0.75 2 0.784 0.75 I). 3 0.778 0.875 ((.4 0.819 0.875 ((.5 0.860 0.875 (I 6 0.787 0.656 (1.7 0.640 0.5 0.460 0.398 ((.9 0.352 0.333 I .0 0.325 0.325 Methods based on document cut-off are particularly vulnerable to small s(tmple fluctuations. The somewhat unusual behaviour of precision in the lower ranges, increasing to 0.860 at recall--0.5 and then decreasing, is probably of this nature. With large samples, one usually finds a monotonic decline. The recall-precision curve using standard recall points with linear interpolation is the furthest removed from the actual data, in the sense that it may contain none of the original averaged precision values. However, it is in a form which permits comparison to other recall-precision curves. For this reason, it is preferable when different systems are being compared. Pessimistic interpolation provides a more conservative view, and, because it is closer to the data, a curve which is usually not so smooth. Statistical inference Techniques of statistical inference are used when the data can be considered a random sample from which generalizations about the population will be made. The particular technique used depends upon the purpose of the research; the scale of the variables.