A critical investigation of recall and precision as measures of retrieval system performance
ACM Transactions on Information Systems (TOIS)
Cumulated gain-based evaluation of IR techniques
ACM Transactions on Information Systems (TOIS)
TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing)
Efficient and effective link analysis with precomputed salsa maps
Proceedings of the 17th ACM conference on Information and knowledge management
Less is more: sampling the neighborhood graph makes SALSA better and faster
Proceedings of the Second ACM International Conference on Web Search and Data Mining
Evaluating the trustworthiness of Wikipedia articles through quality and credibility
Proceedings of the 5th International Symposium on Wikis and Open Collaboration
Finding support sentences for entities
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Entity summarization of news articles
Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval
Improving the multilingual user experience of Wikipedia using cross-language name search
HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
TAER: time-aware entity retrieval-exploiting the past to find relevant entities in news articles
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
Tie-breaking bias: effect of an uncontrolled parameter on information retrieval evaluation
CLEF'10 Proceedings of the 2010 international conference on Multilingual and multimodal information access evaluation: cross-language evaluation forum
Quantifying the trustworthiness of social media content
Distributed and Parallel Databases
Smoothing NDCG metrics using tied scores
Proceedings of the 20th ACM international conference on Information and knowledge management
A framework for the theoretical evaluation of XML retrieval
Journal of the American Society for Information Science and Technology
Evaluations of context-based co-citation searching
Scientometrics
Repeatable and reliable semantic search evaluation
Web Semantics: Science, Services and Agents on the World Wide Web
Machine Learning
| Hi-index | 0.00 |
The Information Retrieval community uses a variety of performance measures to evaluate the effectiveness of scoring functions. In this paper, we show how to adapt six popular measures -- precision, recall, F1, average precision, reciprocal rank, and normalized discounted cumulative gain -- to cope with scoring functions that are likely to assign many tied scores to the results of a search. Tied scores impose only a partial ordering on the results, meaning that there are multiple possible orderings of the result set, each one performing differently. One approach to cope with ties would be to average the performance values across all possible result orderings; but unfortunately, generating result permutations requires super-exponential time. The approach presented in this paper computes precisely the same performance value as the approach of averaging over all permutations, but does so as efficiently as the original, tie-oblivious measures.