IRLIB: Scientific Report No. IRS-13

		SCIENTIFIC REPORT NO. IRS-13 Reports on Evaluation Procedure and Result 1965-1967 Table of Contents
Page
SUMMARY			xi
I. KEEN, E. M.
"Test Environment"
	1. Introduction		I-1
	2. Document Collections and Search Requests		I-1
	3. Relevance Decisions		I-5
	4. Text Experiments		I-11
		A) Experimental Procedures	I-11
		B) Variables Tested	I-13
		C) Vocabularies and Index Language Devices	I-17
	5. Relevance Grade Test Results		I-24
	6. Request and Collection Comparisons		I-28
		A) Request Preparation	I-28
		B) Specific and General Requests	I-32
		C) Collection Comparisons	I-41
References			I-48
II. KEEN, E. M.
"Evaluation Parameters"
	1. Introduction		II-1
	2. Purposes, Viewpoints and Properties of Performance Measures		II-1
	3. Measures for Ranking Systems		II-6
		A) Single Number Measures	II-6
		B) Varying Cut-off Performance Curves	II-7
		C) Comparison of Single Number and Curve Measures	II-13
	4. The Construction of Average Precision Versus Recall Curves		II-18
		A) Averaging Techniques	II-18
		B) Cut-off Techniques	II-21
		C) Extrapolation Techniques for Request Generality Variations	II-31
		D) Extrapolation Techniques for Evaluation of Cluster Searching	II-40
	5. Measures for Varying Relevance Evaluation		II-43
	6. Measures for Varying Generality Comparisons		II-46
	7. Techniques for Dissimilar System Comparisons and Operational Testing		II-51
	8. The Comparison of Specific and General Requests and the Viewpoints of the "higher precisions" and "high recall" user		II-53
	9. The Presentation of Data as Individual Request Merit		II-63
References			II-67
III. KEEN, E. M.
"Search Matching Functions"
	1. Introduction		III-1
	2. Matching Procedures used in Manual, Mechanized and Automated Systems		III-1
		A) Manual Systems	III-1
		B) Mechanized Systems	III-5
		C) Automated Systems	III-8
	3. SMART Test Results - Matching Functions		III-9
		A) Description of Functions	III-9
References			III-58
IV. REITSMA, K. AND SAGALYN, J.
"Correlation Measures"
Abstract			IV-1
1. Introduction			IV-1
2. Weighted versus Logical Description Vectors			IV-2
3. The Correlation Coefficients			IV-5
		A) The Inner Product	IV-6
		B) The Cosine Coefficient	IV-7
		C) The Hypersine Coefficient	IV-7
		D) The Overlap Coefficient	IV-8
		E) The Maron-Kuhns Coefficient	IV-9
		F) The Parker-Rhodes-Needham Coefficient	IV-11
		G) The Stiles Coefficient	IV-13
		H) The Average Coefficient	IV-15
		I) The Reitsma-Sagalyn Coefficient	IV-16
	4. Method of Evaluation		IV-17
	5. Experimental Results		IV-19
	6. Discussion		IV-22
References			IV-26
Tables			IV-27
V. KEEN, E. M.
"Document Length"
	1. Introduction		V-1
	2. SMART Test Comparisons		V-3
	3. Effect of Changes in Document Length		V-4
	4. Test Results		V-13
		A) Abstracts versus Titles	V-14
		B) Abstracts versus Full Text	V-28
		C) Abstracts versus Indexing	V-40
	5. Individual Requests and Discussion of Results		V-48
	6. Conclusions		V-58
References			V-60
VI. KEEN, E. M.
"Suffix Dictionaries"
	1. Introduction		VI-1
	2. Description of Suffix Dictionaries		VI-1
	3. Retrieval Performance Results		VI-4
	4. Performance Analyses		VI-9
	5. Conclusions		VI-20
References			VI-22
VII. KEEN, E. M.
"Thesaurus, Phrase and Hierarchy Dictionaries"
	1. Introduction		VII-1
	2. Description of Thesaurus Dictionaries		VII-1
	3. Description of Phrase Dictionaries		VII-3
	4. Description of Hierarchy Dictionaries		VII-8
	5. Retrieval Performance Results		VII-10
		A) Thesaurus Dictionaries	VII-10
		B) Phrase and Hierarchy Dictionaries	VII-27
	6. Summary of Results		VII-37
	7. Performance Analyses		VII-43
	8. Further Study Required		VII-55
References			VII-58
VIII. DATTOLA, R. T. AND MURRAY, D. M.
"An Experiment in Automatic Thesaurus Construction"
Abstract			VIII-1
	1. Introduction		VIII-1
	2. The Construction Algorithm		VIII-2
		A) Clustering the Document Collection	VIII-3
		B) Formation of Initial Classes	VIII-3
		C) Formation of Merged Classes	VIII-6
		D) Formation of Final Classes	VIII-9
	3. Evaluation		VIII-11
		A) Evaluation of the Classes	VIII-11
		B) Retrieval Evaluation	VIII-15
	4. Analysis of Results		VIII-17
		A) Overlap	VIII-22
		B) Unique Concepts	III-22
		C) Homogenous Concept Classes	VIII-23
		D) Dividing Weights	VIII-24
		E) Cranfield Collection	VIII-24
		F) Comparison of Other Methods	VIII-25
iReferences			VIII-25
IX. LESK, M. E.
"Word-Word Associations in Document Retrieval Systems"
	1. Introduction		IX-1
	2. Method		IX-1
	3. Results		IX-5
	4. Retrieval Experiments		IX-18
	5. Conclusions		IX-50
References			IX-52
X. KEEN, E. M.
"An Analysis of the Documentation Requests"
	1. Introduction		X-1
	2. Request Preparation		X-1
	3. Characteristics of the Requests		X-2
		A) Length	X-2
		B) Important Request Words	X-3
		C) Multiple Need Requests	X-3
		D) Unclear Requests	X-5
		E) Difficult Requests	X-6
	4. Relevance Decisions		X-8
	5. Request Performance		X-9
		A) General Performance Analysis Methods	X-9
		B) Variation in Generality, Length and Concept Frequency	X-10
		C) Comparison of Requests of the Two Preparers	X-20
		D) The Recognition of Important Request Words	X-26
	6. Performance Effectiveness and Search Procedure		X-34
Reference			X-41
Appendix A
	"Recall-Precision Tables"		A-1
Appendix B
	"Original and Modified ADI Queries"		B-1

Retrieval Group home page
IAD home page
Date updated: Friday, 06-Jul-2001 10:22:01 EDT
Date created: Monday, 18-Sept-00

SCIENTIFIC REPORT NO. IRS-13

Reports on Evaluation Procedure and Result 1965-1967

Table of Contents