The Nonstochastic Multiarmed Bandit Problem
SIAM Journal on Computing
Finite-time Analysis of the Multiarmed Bandit Problem
Machine Learning
Introduction to Stochastic Search and Optimization
Introduction to Stochastic Search and Optimization
Policy search using paired comparisons
The Journal of Machine Learning Research
Introduction to Derivative-Free Optimization
Introduction to Derivative-Free Optimization
Hoeffding and Bernstein races for selecting policies in evolutionary direct policy search
ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Yahoo! music recommendations: modeling music ratings with temporal dynamics and item taxonomy
Proceedings of the fifth ACM conference on Recommender systems
Sequential model-based optimization for general algorithm configuration
LION'05 Proceedings of the 5th international conference on Learning and Intelligent Optimization
Random search for hyper-parameter optimization
The Journal of Machine Learning Research
The K-armed dueling bandits problem
Journal of Computer and System Sciences
Hi-index | 0.00 |
In virtually all machine learning applications, hyper-parameter tuning is required to maximize predictive accuracy. Such tuning is computationally expensive, and the cost is further exacerbated by the need for multiple evaluations (via cross-validation or bootstrap) at each configuration setting to guarantee statistically significant results. This paper presents a simple, general technique for improving the efficiency of hyper-parameter tuning by minimizing the number of resampled evaluations at each configuration. We exploit the fact that train-test samples can easily be matched across candidate hyper-parameter configurations. This permits the use of paired hypothesis tests and power analysis that allow for statistically sound early elimination of suboptimal candidates to minimize the number of evaluations. Results on synthetic and real-world datasets demonstrate that our method improves over competitors for discrete parameter settings, and enhances state-of-the-art techniques for continuous parameter settings.