IRE
Information Retrieval Experiment
The pragmatics of information retrieval experimentation
chapter
Jean M. Tague
Butterworth & Company
Karen Sparck Jones
All rights reserved. No part of this publication may be reproduced
or transmitted in any form or by any means, including photocopying
and recording, without the written permission of the copyright holder,
application for which should be addressed to the Publishers. Such
written permission must also be obtained before any part of this
publication is stored in a retrieval system of any nature.
Decision 9: How to analyse the data? 91
.i[OCRerr][OCRerr]igned to recall-- 1. Because of the erratic nature of low recall values for
iI[OCRerr]i[OCRerr] small sample of 4 queries, the latter course is chosen. Similarly, the
.[OCRerr]vcrdge of the precision values for repeating recall values are used, obtaining
` precision of 0.656 for recall=0.692 and 0.398 for recall=0.846. Linear
iliterpolation values [OCRerr]* for the standard recall values r* are obtained from
[OCRerr])t)5crved recall and precision points by using the following formula:
r* -r1
l'[OCRerr] Pi+ r2 - r1 (P2-Pi)
where r1 and r2 are the recall values immediately to the left and to the right
()fr* andp1 andp2 the corresponding precision values. `Pessimistic' precision
v[OCRerr][OCRerr]1ues are those associated with the recorded recall value immediately
((Ilowing (i.e. greater than) the standard one. The values obtained by the two
iiiethods are shown in Table 5.4.
l[OCRerr]ABLE 5.4
.[OCRerr]iondard recall Linear precision Pessimistic precision
1.0 0.75
I 0.892 0.75
2 0.784 0.75
I). 3 0.778 0.875
((.4 0.819 0.875
((.5 0.860 0.875
(I 6 0.787 0.656
(1.7 0.640 0.5
0.460 0.398
((.9 0.352 0.333
I .0 0.325 0.325
Methods based on document cut-off are particularly vulnerable to small
s(tmple fluctuations. The somewhat unusual behaviour of precision in the
lower ranges, increasing to 0.860 at recall--0.5 and then decreasing, is
probably of this nature. With large samples, one usually finds a monotonic
decline.
The recall-precision curve using standard recall points with linear
interpolation is the furthest removed from the actual data, in the sense that
it may contain none of the original averaged precision values. However, it is
in a form which permits comparison to other recall-precision curves. For this
reason, it is preferable when different systems are being compared.
Pessimistic interpolation provides a more conservative view, and, because it
is closer to the data, a curve which is usually not so smooth.
Statistical inference
Techniques of statistical inference are used when the data can be considered
a random sample from which generalizations about the population will be
made. The particular technique used depends upon
the purpose of the research;
the scale of the variables.