IRS13 Evaluation Parameters chapter E. M. Keen Harvard University Gerard Salton Use, reproduction, or publication, in whole or in part, is permitted for any purpose of the United States Government. 11-65 An extension of this type of comparison is the presentation of the magnitudes of the differences in the merit of individual requests. A set of nine hypothetical request results is given in Fig. 36, comparing three options. A table of the numbers of requests preferring options I and II would show that 66-79 prefer option II, and 33.3% prefer option I. However, since the average normalized recall values given in Fig. 36(a) show that options I and II have almost identical merit, it is clear that the three requests preferring I over II do so by quite large amounts, and the six preferring II over I by smaller amounts. The magnitude difference plot in Fig. 36(b) is designed to show this situation visually. The requests favoring each option are arranged in decreasing order of their performance differences accross the plot, and since the areas underneath both curves are nearly equal, this reflects the fact that both options have nearly identical averages. Further, since the option I curve terminates some way short of the option II curve on the x axis, this indicates that more individual requests favor II. Another comparison is given in Fig. 36(c), where option I is seen to be superior to option II both in the averages and in the individual requests.