Measuring relevance judgements
Information Processing and Management: an International Journal
A re-examination of relevance: toward a dynamic, situational definition
Information Processing and Management: an International Journal
Determining the effectiveness of retrieval algorithms
Information Processing and Management: an International Journal
Information retrieval interaction
Information retrieval interaction
A cognitive view of the situational dynamism of user-centered relevance estimation
Journal of the American Society for Information Science - Special issue: relevance research
Journal of the American Society for Information Science - Special issue: relevance research
The formalism of probability theory in IR: a foundation or an encumbrance?
SIGIR '94 Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval
Measuring retrieval effectiveness based on user preference of documents
Journal of the American Society for Information Science
Variations in relevance assessments and the measurement of retrieval effectiveness
Journal of the American Society for Information Science - Special issue: evaluation of information retrieval systems
Information storage and retrieval
Information storage and retrieval
Journal of the American Society for Information Science - Special topic issue on the history of documentation and information science: part II
The probability ranking principle in IR
Readings in information retrieval
How reliable are the results of large-scale information retrieval experiments?
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
Measures of relative relevance and ranked half-life: performance indicators for interactive IR
Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval
The Monte Carlo method and the evaluation of retrieval system performance
Journal of the American Society for Information Science
From highly relevant to not relevant: examining different regions of relevance
Information Processing and Management: an International Journal
When information retrieval measures agree about the relative quality of document rankings
Journal of the American Society for Information Science
Evaluating evaluation measure stability
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
IR evaluation methods for retrieving highly relevant documents
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Evaluation by highly relevant documents
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Modeling score distributions for combining the outputs of search engines
Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval
Information Retrieval
The effect of topic set size on retrieval experiment error
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Liberal relevance criteria of TREC -: counting on negligible documents?
SIGIR '02 Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval
Introduction to Modern Information Retrieval
Introduction to Modern Information Retrieval
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
The text retrieval conferences (TRECS)
TIPSTER '98 Proceedings of a workshop on held at Baltimore, Maryland: October 13-15, 1998
On relevance distributions: Brief Communication
Journal of the American Society for Information Science and Technology
On the reliability of information retrieval metrics based on graded relevance
Information Processing and Management: an International Journal - Special issue: AIRS2005: Information retrieval research in Asia
Toward an understanding of similarity judgments for music digital library evaluation
Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries
Rank-biased precision for measurement of retrieval effectiveness
ACM Transactions on Information Systems (TOIS)
The reliability of metrics based on graded relevance
AIRS'05 Proceedings of the Second Asia conference on Asia Information Retrieval Technology
On effectiveness measures and relevance functions in ranking INEX systems
AIRS'05 Proceedings of the Second Asia conference on Asia Information Retrieval Technology
“THAT’s what i was looking for”: comparing user-rated relevance with search engine rankings
INTERACT'05 Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction
Ranking the NTCIR systems based on multigrade relevance
AIRS'04 Proceedings of the 2004 international conference on Asian Information Retrieval Technology
A classification of IR effectiveness metrics
ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
Experiments on average distance measure
ECIR'06 Proceedings of the 28th European conference on Advances in Information Retrieval
Axiometrics: An Axiomatic Approach to Information Retrieval Effectiveness Metrics
Proceedings of the 2013 Conference on the Theory of Information Retrieval
| Hi-index | 0.00 |
Most common effectiveness measures for information retrieval systems are based on the assumptions of binary relevance (either a document is relevant to a given query or it is not) and binary retrieval (either a document is retrieved or it is not). In this article, these assumptions are questioned, and a new measure named ADM (average distance measure) is proposed, discussed from a conceptual point of view, and experimentally validated on Text Retrieval Conference (TREC) data. Both conceptual analysis and experimental evidence demonstrate ADM's adequacy in measuring the effectiveness of information retrieval systems. Some potential problems about precision and recall are also highlighted and discussed.