IR evaluation methods for retrieving highly relevant documents
SIGIR '00 Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
Rank aggregation methods for the Web
Proceedings of the 10th international conference on World Wide Web
SODA '03 Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms
Efficient similarity search and classification via rank aggregation
Proceedings of the 2003 ACM SIGMOD international conference on Management of data
Retrieval evaluation with incomplete information
Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval
SIAM Journal on Discrete Mathematics
Proceedings of the thirty-ninth annual ACM symposium on Theory of computing
An experimental comparison of click position-bias models
WSDM '08 Proceedings of the 2008 International Conference on Web Search and Data Mining
A new rank correlation coefficient for information retrieval
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Aggregating inconsistent information: Ranking and clustering
Journal of the ACM (JACM)
Proceedings of the Second ACM International Conference on Web Search and Data Mining
On rank correlation and the distance between rankings
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Expected reciprocal rank for graded relevance
Proceedings of the 18th ACM conference on Information and knowledge management
A study of heterogeneity in recommendations for a social music service
Proceedings of the 1st International Workshop on Information Heterogeneity and Fusion in Recommender Systems
Web search solved?: all result rankings the same?
CIKM '10 Proceedings of the 19th ACM international conference on Information and knowledge management
A methodology for evaluating aggregated search results
ECIR'11 Proceedings of the 33rd European conference on Advances in information retrieval
Displacement based unsupervised metric for evaluating rank aggregation
PReMI'11 Proceedings of the 4th international conference on Pattern recognition and machine intelligence
Characterizing web syndication behavior and content
WISE'11 Proceedings of the 12th international conference on Web information system engineering
Reranking search results for sparse queries
Proceedings of the 20th ACM international conference on Information and knowledge management
Learning to aggregate vertical results into web search results
Proceedings of the 20th ACM international conference on Information and knowledge management
Aggregate suppression for enterprise search engines
SIGMOD '12 Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data
Optimal manipulation of voting rules
Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 2
Synthesis ranking with critic resonance
Proceedings of the 3rd Annual ACM Web Science Conference
A comparative study of heterogeneous item recommendations in social systems
Information Sciences: an International Journal
Measuring the Importance of Users in a Social Network Based on Email Communication Patterns
ASONAM '12 Proceedings of the 2012 International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2012)
Editor's Choice Article: Motion-based segmentation of objects using overlapping temporal windows
Image and Vision Computing
Decision making matters: A better way to evaluate trust models
Knowledge-Based Systems
Information quality measurement of medical encoding support based on usability
Computer Methods and Programs in Biomedicine
Efficient multi-keyword ranked query over encrypted data in cloud computing
Future Generation Computer Systems
Entity ranking using click-log information
Intelligent Data Analysis
| Hi-index | 0.00 |
Spearman's footrule and Kendall's tau are two well established distances between rankings. They, however, fail to take into account concepts crucial to evaluating a result set in information retrieval: element relevance and positional information. That is, changing the rank of a highly-relevant document should result in a higher penalty than changing the rank of an irrelevant document; a similar logic holds for the top versus the bottom of the result ordering. In this work, we extend both of these metrics to those with position and element weights, and show that a variant of the Diaconis-Graham inequality still holds - the generalized two measures remain within a constant factor of each other for all permutations. We continue by extending the element weights into a distance metric between elements. For example, in search evaluation, swapping the order of two nearly duplicate results should result in little penalty, even if these two are highly relevant and appear at the top of the list. We extend the distance measures to this more general case and show that they remain within a constant factor of each other. We conclude by conducting simple experiments on web search data with the proposed measures. Our experiments show that the weighted generalizations are more robust and consistent with each other than their unweighted counter-parts.