ALT '08 Proceedings of the 19th international conference on Algorithmic Learning Theory
| Hi-index | 0.00 |
We consider boosting algorithms which maintain a distribution over a set of examples. At each iteration a weak hypothesis is received and the distribution is updated. We motivate these updates as minimizing the relative entropy subject to linear constraints. For example, AdaBoost can be interpreted as constraining the edge of the last hypothesis w.r.t. the updated distribution to be at most γ = 0. In some sense, AdaBoost is "corrective" w.r.t. the last hypothesis. A cleaner boosting method is to be "totally corrective": the edges of all past hypotheses are constrained to be at most γ where γ is suitably adapted. Using new techniques, we prove the same iteration bounds for the totally corrective algorithms as for their corrective versions. Moreover with adaptive γ, the totally corrective algorithms provably maximize the margin. The new techniques include the application of the Generalized Pythagorean Theorem of Bregman divergences and an inequality about relative entropies derived from Pinsker's inequality. We are able to prove iteration bounds for the totally corrective algorithms which optimize the relative entropy w.r.t. the initial or the last distribution as well as for totally corrective algorithms that use the binary relative entropy instead of the relative entropy. Experimentally, the totally corrective versions return smaller convex combinations of weak hypotheses than the corrective ones and are competitive with LP-Boost, a totally corrective boosting algorithm with no regularization, for which there is no iteration bound known.