Accelerating EM for Large Databases
Machine Learning
Toward Optimal Active Learning through Sampling Estimation of Error Reduction
ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Support vector machine active learning with applications to text classification
The Journal of Machine Learning Research
Active Learning with Feedback on Features and Instances
The Journal of Machine Learning Research
Multi-Concept Multi-Modality Active Learning for Interactive Video Annotation
ICSC '07 Proceedings of the International Conference on Semantic Computing
Get another label? improving data quality and data mining using multiple, noisy labelers
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Proactive learning: cost-sensitive active learning with multiple imperfect oracles
Proceedings of the 17th ACM conference on Information and knowledge management
Good learners for evil teachers
ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Efficiently learning the accuracy of labeling sources for selective sampling
Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
Batch Mode Active Learning with Applications to Text Categorization and Image Retrieval
IEEE Transactions on Knowledge and Data Engineering
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Gaussian Processes for Object Categorization
International Journal of Computer Vision
Confidence-based stopping criteria for active learning for data annotation
ACM Transactions on Speech and Language Processing (TSLP)
The Journal of Machine Learning Research
Active learning with sampling by uncertainty and density for data annotations
IEEE Transactions on Audio, Speech, and Language Processing
Active learning in multimedia annotation and retrieval: A survey
ACM Transactions on Intelligent Systems and Technology (TIST)
Cost-Sensitive Active Visual Category Learning
International Journal of Computer Vision
| Hi-index | 0.01 |
In this paper, we focus on obtaining an accurate classifier in active learning, when there are multiple noisy oracles with different and unknown levels of expertise to provide labels for selected instances. We propose a probabilistic model of active learning with multiple noisy oracles (PMActive). Our goal is formulized as to select the most reliable oracle and estimate the actual label on training data. When an instance is selected in every round of active learning, we firstly model the accuracies of individual oracles based on observed noisy labels, and select the most reliable oracle of all to provide a label for the instance. After adding the new instance-label pair into the training set, the actual label of the instance is estimated and used for enhancing the performance of the current classifier. The experimental results indicate that the PMActive method can work with different noise levels of oracles. Compared with the baselines which are commonly used in this area of active learning, the PMActive method is superior in obtaining a more accurate classifier.