IRE
Information Retrieval Experiment
The pragmatics of information retrieval experimentation
chapter
Jean M. Tague
Butterworth & Company
Karen Sparck Jones
All rights reserved. No part of this publication may be reproduced
or transmitted in any form or by any means, including photocopying
and recording, without the written permission of the copyright holder,
application for which should be addressed to the Publishers. Such
written permission must also be obtained before any part of this
publication is stored in a retrieval system of any nature.
I
82 The pragmatics of information retrieval experimentation I
+
Dayl Day?
Order 1 2 3 4 5 6 7 8 9 10
Position a b a b a b a b a b a b a b a b a b a b
Requests 1 13 215 414 5 11 3 12 10 17 620 816 7 18 9 19
1 A ,D1 E C A C E D
Ii,
L[OCRerr]2 B [OCRerr]EI A D B D A E
__ AED
[OCRerr]3 C IAI B E C E B A
[OCRerr]
L
a[OCRerr]4 D C A D A 0 B
[OCRerr],,
5 E D B E B C
Requests 202716301826 1728 1929 1123 1225 1124 1521 1322
6 A E D A B C
[OCRerr] ;E[OCRerr]AE
7 B A E B C D
;Q;j)ED)0 A[OCRerr]D
LU 8 C B A C D E
r /AIIAB
O[OCRerr]9 D C B D E A
(1) [OCRerr]B1[OCRerr]C
10 E C E B
Requests 21 33 22 35 24 34 25 31 23 32 30 37 26 40 28 36 27 38 29 39
11 A C A D
ti, C(½%½E
G, [OCRerr]II[OCRerr][OCRerr]"
L 12 B E
u 13 C
L E AECDB¼;MFA:D;: ED A
r E([OCRerr]½[OCRerr]B
14 D AB[OCRerr] B
[OCRerr]
15 BA¼[OCRerr]CD C
Requests 4047 3650 3846 37483949 3143 3245 3444 3541 3342
16 A E D A B C
Ln [OCRerr]ThEA
L 17 B E B C D
CEO A j[OCRerr][OCRerr]B
[OCRerr] ,`[OCRerr],.,,
18 C D/ B A C D E
U
L
[OCRerr] 19 D A C B D E A
& D C E
[OCRerr]
"' 20 E B Affi'CB; B
Requests 413 425 444 451 432 507 4610 486 478 499
21 A C A[OCRerr]C1B,[OCRerr]E[OCRerr]
[OCRerr] 22 B D B[OCRerr]DICJIA\
[OCRerr] iilii[OCRerr]
L 23 C
_____ __________________ /0
_ ;&:;I __
a
U EDLLAELJDCB)
24 D
25 E
Figure 5.2. Incomplete block experimental design used in EPSILON
test (from Keen and Wheatley). Indexes (A-E), blocks (1-10) are
Latin squares, pairs of blocks (1/2, 3/4, etc.) are balanced
The number of queries in previous information retrieval tests seems to
vary from 15 to 300, with values in the range 50 to 100 being most common.
Of course, to assess these numbers, one needs to know if queries are
completely or incompletely crossed with other factors.
I