Communications of the ACM
COLT '92 Proceedings of the fifth annual workshop on Computational learning theory
C4.5: programs for machine learning
C4.5: programs for machine learning
A sequential algorithm for training text classifiers
SIGIR '94 Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval
Improving Generalization with Active Learning
Machine Learning - Special issue on structured connectionist systems
A simple, fast, and effective rule learner
AAAI '99/IAAI '99 Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence
Active learning using adaptive resampling
Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining
Machine Learning
Machine Learning
The Case against Accuracy Estimation for Comparing Induction Algorithms
ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Query Learning Strategies Using Boosting and Bagging
ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Learning and Making Decisions When Costs and Probabilities are Both Unknown
Learning and Making Decisions When Costs and Probabilities are Both Unknown
Active learning with statistical models
Journal of Artificial Intelligence Research
Class Probability Estimation and Cost-Sensitive Classification Decisions
ECML '02 Proceedings of the 13th European Conference on Machine Learning
Active Sampling for Feature Selection
ICDM '03 Proceedings of the Third IEEE International Conference on Data Mining
Diverse ensembles for active learning
ICML '04 Proceedings of the twenty-first international conference on Machine learning
Efficiently exploring architectural design spaces via predictive modeling
Proceedings of the 12th international conference on Architectural support for programming languages and operating systems
Efficient sampling of training set in large and noisy multimedia data
ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
Active learning for logistic regression: an evaluation
Machine Learning
Efficient architectural design space exploration via predictive modeling
ACM Transactions on Architecture and Code Optimization (TACO)
Customer targeting models using actively-selected web content
Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
ECML '07 Proceedings of the 18th European conference on Machine Learning
Journal of Data and Information Quality (JDIQ)
Learning when training data are costly: the effect of class distribution on tree induction
Journal of Artificial Intelligence Research
Active cost-sensitive learning
IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Accurate and efficient processor performance prediction via regression tree based modeling
Journal of Systems Architecture: the EUROMICRO Journal
Cost-minimising strategies for data labelling: optimal stopping and active learning
FoIKS'08 Proceedings of the 5th international conference on Foundations of information and knowledge systems
Architecture performance prediction using evolutionary artificial neural networks
Evo'08 Proceedings of the 2008 conference on Applications of evolutionary computing
Active sampling for knowledge discovery from biomedical data
PKDD'05 Proceedings of the 9th European conference on Principles and Practice of Knowledge Discovery in Databases
Active learning for probability estimation using jensen-shannon divergence
ECML'05 Proceedings of the 16th European conference on Machine Learning
Active learning for probabilistic neural networks
ICNC'05 Proceedings of the First international conference on Advances in Natural Computation - Volume Part I
Musical sound recognition by active learning PNN
MRCS'06 Proceedings of the 2006 international conference on Multimedia Content Representation, Classification and Security
Model guided adaptive design and analysis in computer experiment
Statistical Analysis and Data Mining
| Hi-index | 0.00 |
For many supervised learning tasks it is very costly to produce training data with class labels. Active learning acquires data incrementally, at each stage using the model learned so far to help identify especially useful additional data for labeling. Existing empirical active learning approaches have focused on learning classifiers. However, many applications require estimations of the probability of class membership, or scores that can be used to rank new cases. We present a new active learning method for class probability estimation (CPE) and ranking. BOOTSTRAP-LV selects new data for labeling based on the variance in probability estimates, as determined by learning multiple models from bootstrap samples of the existing labeled data. We show empirically that the method reduces the number of data items that must be labeled, across a wide variety of data sets. We also compare BOOTSTRAP-LV with UNCERTAINTY SAMPLING, an existing active learning method designed to maximize classification accuracy. The results show that BOOTSTRAP-LV dominates for CPE. Surprisingly it also often is preferable for accelerating simple accuracy maximization.