Combining the evidence of multiple query representations for information retrieval
TREC-2 Proceedings of the second conference on Text retrieval conference
Rank aggregation methods for the Web
Proceedings of the 10th international conference on World Wide Web
Proceedings of the 16th international conference on World Wide Web
SIGIR '07 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval
A new rank correlation coefficient for information retrieval
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
On rank correlation and the distance between rankings
Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Generalized distances between rankings
Proceedings of the 19th international conference on World wide web
Hi-index | 0.00 |
Rank Aggregation is the problem of aggregating ranks given by various experts to a set of entities. In context of web, it has applications like building metasearch engines, combining user preferences etc. For many of these applications, it is difficult to get labeled data and the aggregation algorithms need to be evaluated against unsupervised evaluation metrics. We consider the Kendall-Tau unsupervised metric which is widely used for evaluating rank aggregation task. Kendall Tau distance between two permutations is defined as the number of pairwise inversions among the permutations. The original Kendall Tau distance treats each inversion equally, irrespective of the differences in rank positions of the inverted items. In this work, we propose a variant of Kendall-Tau distance that takes into consideration this difference in rank positions. We study, examine and compare various available supervised as well as unsupervised metrics with the proposed metric. We experimentally demonstrate that our modification in Kendall Tau Distance makes it potentially better than other available unsupervised metrics for evaluating aggregated ranking.