Learning from multiple annotators with Gaussian processes

Authors:
Perry Groot;Adriana Birlutiu;Tom Heskes
Affiliations:
Technical University Eindhoven, Dept. of Electrical Engineering, Control Systems, Eindhoven, The Netherlands;Radboud University Nijmegen, Intelligent Systems, Nijmegen, The Netherlands;Radboud University Nijmegen, Intelligent Systems, Nijmegen, The Netherlands
Venue:
ICANN'11 Proceedings of the 21st international conference on Artificial neural networks - Volume Part II
Year:
2011

Citing 8
Cited 2

Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)

Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)
Get another label? improving data quality and data mining using multiple, noisy labelers

Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Veritas: Combining Expert Opinions without Labeled Data

ICTAI '08 Proceedings of the 2008 20th IEEE International Conference on Tools with Artificial Intelligence - Volume 01
Good learners for evil teachers

ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Supervised learning from multiple experts: whom to trust when everyone lies a bit

ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Learning From Crowds

The Journal of Machine Learning Research
Regression Learning with Multiple Noisy Oracles

Proceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence

Learning from multiple annotators: Distinguishing good from random labelers

Pattern Recognition Letters
Selective sampling and active learning from single and multiple teachers

The Journal of Machine Learning Research

Quantified Score

Hi-index	0.00

Visualization

Abstract

In many supervised learning tasks it can be costly or infeasible to obtain objective, reliable labels. We may, however, be able to obtain a large number of subjective, possibly noisy, labels from multiple annotators. Typically, annotators have different levels of expertise (i.e., novice, expert) and there is considerable diagreement among annotators. We present a Gaussian process (GP) approach to regression with multiple labels but no absolute gold standard. The GP framework provides a principled non-parametric framework that can automatically estimate the reliability of individual annotators from data without the need of prior knowledge. Experimental results show that the proposed GP multi-annotator model outperforms models that either average the training data or weigh individually learned single-annotator models.