IRE Simulation, and simulation experiments chapter Michael D. Heine Butterworth & Company Karen Sparck Jones All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder, application for which should be addressed to the Publishers. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature. Some previous work in simulation applied to information retrieval 195 operational Systems (where Cooper was concerned with more hypothetical data) this objective does not appear to have met with complete success, since (a) operational data for a full validation of the simulation model was not obtainable, and (b) the data that was obtainable from existing small test collections was either inadequate or inapplicable (p. 11). The main goal of preparing an information retrieval system simulation appropriate to operational retrieval systems appears therefore to be far from complete, since even if a model incorporating valid real data in all significant components could be found, there would still remain the problem of `designing this in' to a larger model taking into account the motivations of users and supporting agencies, as discussed by Reilly, and Baker and Nance. If valid data cannot be obtained (which seems unlikely), then there is a limitation here in princiole to the usefulness of the simulation approach. Further discussion The general matter of the appropriateness of incorporating test-collection based data into a simulation model deserves careful examination. A prior question is whether research on such collections should be regarded as `simulation work'[OCRerr]a point touched on in the introduction. Given the immense amount of valuable work that has been done using (say) the Cranfield experimental data32, by researchers not involved in the actual acquisition of that data: for example Salton3, Sparck Jones33 or van Rusbergen and Sparck Jones34, it seems reasonable to regard such work as `simulation'. It is oriented towards optimization of a process (information retrieval) through intervention of some kind in it, and is concerned with data already found. It is a substitute for further data acquisition achieved through the manipulation of experimental variables, i.e. manipulation of the `apparatus'. It has become clear from such work that the experimental data incorporated into the test collections are in fact an indispensable input to analyses of a very wide range, as discussed in detail elsewhere in this volume, and that such analyses were not initially objectives of the experimental work. They have clearly `spun off' it. In the sense that such theoretical work uses both the experimental data as input, and uses some of the representations of system components recognized in the experimental work, it is reasonable if somewhat arbitrary to regard it as simulation work. This does not perhaps take us very far, but it points to the conceptual barrier between theoretical work (simulation work in the broader sense) and experimental work as a hazy and rather unsatisfactory one: theory involves constructs; experiment involves apparatus; both manipulate and control data. Of much greater concern, since it is not just a semantic matter, is the question of whether it is legitimate to use experimental data (an archive of which may be stored as a `test collection') in a simulation study when the data was obtained using questionable methodology. A clear example here is the employment in the experiment of a requirement that `relevance' be judged in relation to a `question' where the latter is defined to be a solely verbal construct having no explicit relationship to a context of information need. (The question might be a sentence or paragraph in English, for example.) Here the relevance judgements that appear as experimental data are made with ambiguous terms of reference. Is the arbiter of relevance to hypothesize