Optimizing nDCG gains by minimizing effect of label inconsistency

Authors:
Pavel Metrikov;Virgil Pavlu;Javed A. Aslam
Affiliations:
Northeastern University, Boston, MA;Northeastern University, Boston, MA;Northeastern University, Boston, MA
Venue:
ECIR'13 Proceedings of the 35th European conference on Advances in Information Retrieval
Year:
2013

Citing 2
Cited 1

Evaluating evaluation metrics based on the bootstrap

SIGIR '06 Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
Relevance assessment: are judges exchangeable and does it matter

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

ECIR 2013: 35th european conference on information retrieval

ACM SIGIR Forum

Quantified Score

Hi-index	0.00

Visualization

Abstract

We focus on nDCG choice of gains, and in particular on the fracture between large differences in exponential gains of high relevance labels and the not-so-small confusion, or inconsistency, between these labels in data. We show that better gains can be derived from data by measuring the label inconsistency, to the point that virtually indistinguishable labels correspond to equal gains. Our derived optimal gains make a better nDCG objective for training Learning to Rank algorithms.