Cranking: Combining Rankings Using Conditional Probability Models on Permutations
ICML '02 Proceedings of the Nineteenth International Conference on Machine Learning
Learning to rank: from pairwise approach to listwise approach
Proceedings of the 24th international conference on Machine learning
Unsupervised rank aggregation with distance-based models
Proceedings of the 25th international conference on Machine learning
Listwise approach to learning to rank: theory and algorithm
Proceedings of the 25th international conference on Machine learning
Get another label? improving data quality and data mining using multiple, noisy labelers
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
The Journal of Machine Learning Research
Learning from multiple annotators: Distinguishing good from random labelers
Pattern Recognition Letters
Hi-index | 0.00 |
Learning to rank has received great attention in recent years as it plays a crucial role in information retrieval. The existing concept of learning to rank assumes that each training sample is associated with an instance and a reliable label. However, in practice, this assumption does not necessarily hold true. This study focuses on the learning to rank when each training instance is labeled by multiple annotators that may be unreliable. In such a scenario, no accurate labels can be obtained. This study proposes two learning approaches. One is to simply estimate the ground truth first and then to learn a ranking model with it. The second approach is a maximum likelihood learning approach which estimates the ground truth and learns the ranking model iteratively. The two approaches have been tested on both synthetic and real-world data. The results reveal that the maximum likelihood approach outperforms the first approach significantly and is comparable of achieving results with the learning model considering reliable labels. Further more, both the approaches have been applied for ranking the Web visual clutter.