Learning with an unreliable teacher
Pattern Recognition
A view of the EM algorithm that justifies incremental, sparse, and other variants
Learning in graphical models
A Simple Approach to Ordinal Classification
EMCL '01 Proceedings of the 12th European Conference on Machine Learning
Get another label? improving data quality and data mining using multiple, noisy labelers
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Multiple-Instance Learning Improves CAD Detection of Masses in Digital Mammography
IWDM '08 Proceedings of the 9th international workshop on Digital Mammography
Veritas: Combining Expert Opinions without Labeled Data
ICTAI '08 Proceedings of the 2008 20th IEEE International Conference on Tools with Artificial Intelligence - Volume 01
Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business
Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business
Good learners for evil teachers
ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Supervised learning from multiple experts: whom to trust when everyone lies a bit
ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Efficiently learning the accuracy of labeling sources for selective sampling
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Quality management on Amazon Mechanical Turk
Proceedings of the ACM SIGKDD Workshop on Human Computation
Bayesian knowledge corroboration with logical rules and user feedback
ECML PKDD'10 Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part II
CoBayes: bayesian knowledge corroboration with assessors of unknown areas of expertise
Proceedings of the fourth ACM international conference on Web search and data mining
MCV'10 Proceedings of the 2010 international MICCAI conference on Medical computer vision: recognition techniques and applications in medical imaging
Cooks or cobblers?: crowd creativity through combination
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
User reputation in a comment rating environment
Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining
Feature selection in crowd creativity
FAC'11 Proceedings of the 6th international conference on Foundations of augmented cognition: directing the future of adaptive systems
Learning from multiple annotators with Gaussian processes
ICANN'11 Proceedings of the 21st international conference on Artificial neural networks - Volume Part II
Generalized agreement statistics over fixed group of experts
ECML PKDD'11 Proceedings of the 2011 European conference on Machine learning and knowledge discovery in databases - Volume Part III
Get your jokes right: ask the crowd
MEDI'11 Proceedings of the First international conference on Model and data engineering
Pushing the boundaries of crowd-enabled databases with query-driven schema expansion
Proceedings of the VLDB Endowment
Eliminating spammers and ranking annotators for crowdsourced labeling tasks
The Journal of Machine Learning Research
Peer prediction without a common prior
Proceedings of the 13th ACM Conference on Electronic Commerce
Learning to rank under multiple annotators
IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence - Volume Volume Two
Learning from crowds in the presence of schools of thought
Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining
Whom to ask?: jury selection for decision making tasks on micro-blog services
Proceedings of the VLDB Endowment
Crowd IQ: measuring the intelligence of crowdsourcing platforms
Proceedings of the 3rd Annual ACM Web Science Conference
Robust detection of comment spam using entropy rate
Proceedings of the 5th ACM workshop on Security and artificial intelligence
Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations
Language Resources and Evaluation
The acoustic emotion gaussians model for emotion-based music annotation and retrieval
Proceedings of the 20th ACM international conference on Multimedia
Map to humans and reduce error: crowdsourcing for deduplication applied to digital libraries
Proceedings of the 21st ACM international conference on Information and knowledge management
Dynamic probabilistic CCA for analysis of affective behaviour
ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part VII
Label-Noise robust logistic regression and its applications
ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Mining anatomical, physiological and pathological information from medical images
ACM SIGKDD Explorations Newsletter
Pairwise ranking aggregation in a crowdsourced setting
Proceedings of the sixth ACM international conference on Web search and data mining
An internet-scale idea generation system
ACM Transactions on Interactive Intelligent Systems (TiiS) - Special section on internet-scale human problem solving and regular papers
An introduction to crowdsourcing for language and multimedia technology research
PROMISE'12 Proceedings of the 2012 international conference on Information Retrieval Meets Information Visualization
WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 02
Tagging human activities in video by crowdsourcing
Proceedings of the 3rd ACM conference on International conference on multimedia retrieval
An online cost sensitive decision-making method in crowdsourcing systems
Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
Efficient crowdsourcing for multi-class labeling
Proceedings of the ACM SIGMETRICS/international conference on Measurement and modeling of computer systems
Trust-based fusion of untrustworthy information in crowdsourcing applications
Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Towards a generic framework for trustworthy spatial crowdsourcing
Proceedings of the 12th International ACM Workshop on Data Engineering for Wireless and Mobile Acess
A transfer learning based framework of crowd-selection on twitter
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Information verification during natural disasters
Proceedings of the 22nd international conference on World Wide Web companion
Aggregating crowdsourced binary ratings
Proceedings of the 22nd international conference on World Wide Web
Pricing mechanisms for crowdsourcing markets
Proceedings of the 22nd international conference on World Wide Web
A threshold method for imbalanced multiple noisy labeling
Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining
Learning from multiple annotators: Distinguishing good from random labelers
Pattern Recognition Letters
Proceedings of the International Workshop on Video and Image Ground Truth in Computer Vision Applications
Selective sampling and active learning from single and multiple teachers
The Journal of Machine Learning Research
Instant foodie: predicting expert ratings from grassroots
Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
GeoTruCrowd: trustworthy query answering with spatial crowdsourcing
Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems
Computer Vision and Image Understanding
A lightweight combinatorial approach for inferring the ground truth from multiple annotators
MLDM'13 Proceedings of the 9th international conference on Machine Learning and Data Mining in Pattern Recognition
Accurate integration of crowdsourced labels using workers' self-reported confidence scores
IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Trust, but verify: predicting contribution quality for knowledge base construction and curation
Proceedings of the 7th ACM international conference on Web search and data mining
Learning classification models from multiple experts
Journal of Biomedical Informatics
Expert Systems with Applications: An International Journal
Quizz: targeted crowdsourcing with a billion (potential) users
Proceedings of the 23rd international conference on World wide web
Community-based bayesian aggregation models for crowdsourcing
Proceedings of the 23rd international conference on World wide web
The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing
Proceedings of the 23rd international conference on World wide web
Spontaneous facial expression recognition: A robust metric learning approach
Pattern Recognition
Repeated labeling using multiple noisy labelers
Data Mining and Knowledge Discovery
Mixtures of biased sentiment analysers
Advances in Data Analysis and Classification
Hi-index | 0.00 |
For many supervised learning tasks it may be infeasible (or very expensive) to obtain objective and reliable labels. Instead, we can collect subjective (possibly noisy) labels from multiple experts or annotators. In practice, there is a substantial amount of disagreement among the annotators, and hence it is of great practical interest to address conventional supervised learning problems in this scenario. In this paper we describe a probabilistic approach for supervised learning when we have multiple annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates the different experts and also gives an estimate of the actual hidden labels. Experimental results indicate that the proposed method is superior to the commonly used majority voting baseline.