Approximation stability and boosting

Authors:
Wei Gao;Zhi-Hua Zhou
Affiliations:
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China;National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Venue:
ALT'10 Proceedings of the 21st international conference on Algorithmic learning theory
Year:
2010

Citing 11
Cited 1

Game theory, on-line prediction and boosting

COLT '96 Proceedings of the ninth annual conference on Computational learning theory
A decision-theoretic generalization of on-line learning and an application to boosting

Journal of Computer and System Sciences - Special issue: 26th annual ACM symposium on the theory of computing & STOC'94, May 23–25, 1994, and second annual Europe an conference on computational learning theory (EuroCOLT'95), March 13–15, 1995
Scale-sensitive dimensions, uniform convergence, and learnability

Journal of the ACM (JACM)
Algorithmic stability and sanity-check bounds for leave-one-out cross-validation

Neural Computation
Prediction games and arcing algorithms

Neural Computation
Soft Margins for AdaBoost

Machine Learning
Stability and generalization

The Journal of Machine Learning Research
How boosting the margin can also boost classifier complexity

ICML '06 Proceedings of the 23rd international conference on Machine learning
Some Theory for Generalized Boosting Algorithms

The Journal of Machine Learning Research
Evidence Contrary to the Statistical View of Boosting

The Journal of Machine Learning Research
Almost-everywhere algorithmic stability and generalization error

UAI'02 Proceedings of the Eighteenth conference on Uncertainty in artificial intelligence

Uniform convergence, stability and learnability for ranking problems

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence

Quantified Score

Hi-index	0.00

Visualization

Abstract

Stability has been explored to study the performance of learning algorithms in recent years and it has been shown that stability is sufficient for generalization and is sufficient and necessary for consistency of ERM in the general learning setting. Previous studies showed that AdaBoost has almost-everywhere uniform stability if the base learner has L1 stability. The L1 stability, however, is too restrictive and we show that AdaBoost becomes constant learner if the base learner is not real-valued learner. Considering that AdaBoost is mostly successful as a classification algorithm, stability analysis for AdaBoost when the base learner is not real-valued learner is an important yet unsolved problem. In this paper, we introduce the approximation stability and prove that approximation stability is sufficient for generalization, and sufficient and necessary for learnability of AERM in the general learning setting. We prove that AdaBoost has approximation stability and thus has good generalization, and an exponential bound for AdaBoost is provided.