IRE The pragmatics of information retrieval experimentation chapter Jean M. Tague Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. 100 The pragmatics of information retrieval experimentation 5.10 Decision 10: How to present results? Information retrieval experiments should be written up as experiments. This rather obvious recommendation is not always followed in practice. In many reports, one does not realize an experiment has been performed until half way through the paper. The first part is all background. The various aspects of an experiment are generally described in the following order: Purpose of the experiment. Background for the experiment. Methodology. Presentation of results. Summary and conclusions. The purpose of the experiment should be described both in general and specific terms, i.e. the general problem or hypothesis being investigated and its realization, for this experiment, in terms of operational variables. The background section should provide justification for the experiment. What previous work has been carried out in this area? Why is the present study needed? What led the investigator to undertake the work? Only references that specifically relate to the problem under study should be included. Methodology can usually be subdivided into two sections: the test environment and the test procedures. The environment refers to the characteristics of the documents, document surrogates, queries, users, searchers, equipment, etc., used in the test. These should be characterized in detail, as the generality of the results depend on these aspects. Procedures relate to the actual methods used to select the sample (experimental design), run the experiment, collect the data, and analyse the results. Procedures should be described in sufficient detail that another experimenter can repeat the experiment. However, aspects which have been described previously, in generally accessible documents (i.e. not in private communications), such as search algorithms or statistical tests, should simply be referenced. In the results section, the investigator attempts to summarize verbally what the experimental results have shown, not just present pages of tables. Detailed computations or mathematical derivations should be relegated to an appendix and their conclusions only incorporated in the text. Similarly, detailed results and analyses, such as a query by query failure analysis, should also be in the appendix. The final section should serve to review, reiterate, and summarize what has gone before. Remember that this section is all many people read! Some small but important matters remain to mention. Symbols should not be introduced without precise definition. Even conventional symbols can have several interpretations: for example, it can represent either a mathematical constant, or a product operation, or system precision. Both the horizontal and vertical axes in graphs should be labelled and the scale indicated. Unless results are really voluminous, graphs should be accompanied by tables showing the specific values used to construct them. All graphs and other figures should have legends. Other investigators may