Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms

Authors:
Chris Thornton;Frank Hutter;Holger H. Hoos;Kevin Leyton-Brown
Affiliations:
University of British Columbia, Vancouver, BC, Canada;University of British Columbia, Vancouver, BC, Canada;University of British Columbia, Vancouver, BC, Canada;University of British Columbia, Vancouver, BC, Canada
Venue:
Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining
Year:
2013

Citing 19
Cited 0

Efficient Global Optimization of Expensive Black-Box Functions

Journal of Global Optimization
Model Selection for Small Sample Regression

Machine Learning
A perspective view and survey of meta-learning

Artificial Intelligence Review
Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results

Machine Learning
Meta-Learning by Landmarking Various Learning Algorithms

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
A Model Selection Criterion for Classification: Application to HMM Topology Optimization

ICDAR '03 Proceedings of the Seventh International Conference on Document Analysis and Recognition - Volume 1
Gradient-Based Optimization of Hyperparameters

Neural Computation
On Model Selection Consistency of Lasso

The Journal of Machine Learning Research
A novel LS-SVMs hyper-parameter selection based on particle swarm optimization

Neurocomputing
Model selection for the LS-SVM. Application to handwriting recognition

Pattern Recognition
A study of cross-validation and bootstrap for accuracy estimation and model selection

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
The WEKA data mining software: an update

ACM SIGKDD Explorations Newsletter
ParamILS: an automatic algorithm configuration framework

Journal of Artificial Intelligence Research
PyBrain

The Journal of Machine Learning Research
Nonlinear regression model generation using hyperparameter optimization

Computers & Mathematics with Applications
Sequential model-based optimization for general algorithm configuration

LION'05 Proceedings of the 5th international conference on Learning and Intelligent Optimization
Random search for hyper-parameter optimization

The Journal of Machine Learning Research
Selecting classification algorithms with active testing

MLDM'12 Proceedings of the 8th international conference on Machine Learning and Data Mining in Pattern Recognition
Parallel algorithm configuration

LION'12 Proceedings of the 6th international conference on Learning and Intelligent Optimization

Quantified Score

Hi-index	0.00

Visualization

Abstract

Many different machine learning algorithms exist; taking into account each algorithm's hyperparameters, there is a staggeringly large number of possible alternatives overall. We consider the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that attacks these issues separately. We show that this problem can be addressed by a fully automated approach, leveraging recent innovations in Bayesian optimization. Specifically, we consider a wide range of feature selection techniques (combining 3 search and 8 evaluator methods) and all classification approaches implemented in WEKA's standard distribution, spanning 2 ensemble methods, 10 meta-methods, 27 base classifiers, and hyperparameter settings for each classifier. On each of 21 popular datasets from the UCI repository, the KDD Cup 09, variants of the MNIST dataset and CIFAR-10, we show classification performance often much better than using standard selection and hyperparameter optimization methods. We hope that our approach will help non-expert users to more effectively identify machine learning algorithms and hyperparameter settings appropriate to their applications, and hence to achieve improved performance.