Blind Men and Elephants: Six Approaches to TREC data

Authors:
David Banks;Paul Over;Nien-Fan Zhang
Affiliations:
National Institute of Standards and Technology (NIST), Gaithersburg, MD 20899;National Institute of Standards and Technology (NIST), Gaithersburg, MD 20899;National Institute of Standards and Technology (NIST), Gaithersburg, MD 20899
Venue:
Information Retrieval
Year:
1999

Citing 4
Cited 23

Overview of the second text retrieval conference (TREC-2)

TREC-2 Proceedings of the second conference on Text retrieval conference
Visualization and scaling of TREC topic document sets

Information Processing and Management: an International Journal
SAS/Iml Software, Changes and Enhancements through Release 6. 11

SAS/Iml Software, Changes and Enhancements through Release 6. 11
Clustering Algorithms

Clustering Algorithms

Latent semantic space: iterative scaling improves precision of inter-document similarity measurement

SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Report on TREC-9

ACM SIGIR Forum
The effect of topic set size on retrieval experiment error

SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Analysis of performance variation using query expansion

Journal of the American Society for Information Science and Technology
System Performance and Natural Language Expression of Information Needs

Information Retrieval
Statistical principal components analysis for retrieval experiments: Research Articles

Journal of the American Society for Information Science and Technology
Test theory for assessing IR test collections

SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
Test theory for evaluating reliability of IR test collections

Information Processing and Management: an International Journal
On test collections for adaptive information retrieval

Information Processing and Management: an International Journal
If I Had a Million Queries

ECIR '09 Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval
Topic set size redux

Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
A few good topics: Experiments in topic set reduction for retrieval evaluation

ACM Transactions on Information Systems (TOIS)
Empirical justification of the gain and discount function for nDCG

Proceedings of the 18th ACM conference on Information and knowledge management
Evaluation in context

ECDL'09 Proceedings of the 13th European conference on Research and advanced technology for digital libraries
Toward enabling users to visually evaluate the effectiveness of different search methods

Journal of Web Engineering
To re-rank or to re-query: can visual analytics solve this dilemma?

CLEF'11 Proceedings of the Second international conference on Multilingual and multimodal information access evaluation
Fusing different information retrieval systems according to query-topics: a study based on correlation in information retrieval systems and TREC topics

Information Retrieval
On smoothing average precision

ECIR'12 Proceedings of the 34th European conference on Advances in Information Retrieval
On per-topic variance in IR evaluation

SIGIR '12 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval
Visual interactive failure analysis: supporting users in information retrieval evaluation

Proceedings of the 4th Information Interaction in Context Symposium
DIRECTions: design and specification of an IR evaluation infrastructure

CLEF'12 Proceedings of the Third international conference on Information Access Evaluation: multilinguality, multimodality, and visual analytics
On the measurement of test collection reliability

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
Bias-variance analysis in estimating true query model for information retrieval

Information Processing and Management: an International Journal

Quantified Score

Hi-index	0.00

Visualization

Abstract

The paper reviews six recent efforts to better understand performancemeasurements on information retrieval (IR) systems within theframework of the Text REtrieval Conferences (TREC): analysis ofvariance, cluster analyses, rank correlations, beadplots,multidimensional scaling, and item response analysis. None of thiswork has yielded any substantial new insights. Prospects thatadditional work along these lines will yield more interesting resultsvary but are in general not promising. Some suggestions are made forpaying greater attention to richer descriptions of IR system behaviorbut within smaller, better controlled settings.