Linear Programming Boosting via Column Generation

Authors:
Ayhan Demiriz;Kristin P. Bennett;John Shawe-Taylor
Affiliations:
Department of Decision Sciences and Eng. Systems, Rensselaer Polytechnic Institute, Troy, NY 12180, USA. demira@rpi.edu;Department of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY 12180 USA while visiting Microsoft Research, Redmond, WA, USA. bennek@rpi.edu;Department of Computer Science, Royal Holloway, University of London, Egham, Surrey TW20 0EX, UK. jst@cs.rhul.ac.uk
Venue:
Machine Learning
Year:
2002

Citing 13
Cited 66

A training algorithm for optimal margin classifiers

COLT '92 Proceedings of the fifth annual workshop on Computational learning theory
Support-Vector Networks

Machine Learning
Boosting in the limit: maximizing the margin of learned ensembles

AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Combining support vector and mathematical programming methods for classification

Advances in kernel methods
Prediction games and arcing algorithms

Neural Computation
Improved Boosting Algorithms Using Confidence-rated Predictions

Machine Learning - The Eleventh Annual Conference on computational Learning Theory
Semi-supervised support vector machines

Proceedings of the 1998 conference on Advances in neural information processing systems II
An introduction to support Vector Machines: and other kernel-based learning methods

An introduction to support Vector Machines: and other kernel-based learning methods
Learning in Neural Networks: Theoretical Foundations

Learning in Neural Networks: Theoretical Foundations
An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

Machine Learning
Duality and Geometry in SVM Classifiers

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
A Column Generation Algorithm For Boosting

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Bagging, boosting, and C4.S

AAAI'96 Proceedings of the thirteenth national conference on Artificial intelligence - Volume 1

Constructing Boosting Algorithms from SVMs: An Application to One-Class Classification

IEEE Transactions on Pattern Analysis and Machine Intelligence
An introduction to boosting and leveraging

Advanced lectures on machine learning
Column-generation boosting methods for mixture of kernels

Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
The Dynamics of AdaBoost: Cyclic Behavior and Convergence of Margins

The Journal of Machine Learning Research
Semi-Supervised Mixture of Kernels via LPBoost Methods

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Parameter estimation for statistical parsing models: theory and practice of distribution-free methods

New developments in parsing technology
Ensemble Pruning Via Semi-definite Programming

The Journal of Machine Learning Research
Iterative RELIEF for Feature Weighting: Algorithms, Theories, and Applications

IEEE Transactions on Pattern Analysis and Machine Intelligence
Robust Loss Functions for Boosting

Neural Computation
Increasing the Robustness of Boosting Algorithms within the Linear-programming Framework

Journal of VLSI Signal Processing Systems
Object categorization

Foundations and Trends® in Computer Graphics and Vision
A multiple kernel support vector machine scheme for feature selection and rule extraction from gene expression data of cancer tissue

Artificial Intelligence in Medicine
Recognizing cars in aerial imagery to improve orthophotos

Proceedings of the 15th annual ACM international symposium on Advances in geographic information systems
Multi-group support vector machines with measurement costs: A biobjective approach

Discrete Applied Mathematics
Support Vector Machinery for Infinite Ensemble Learning

The Journal of Machine Learning Research
Partial least squares regression for graph mining

Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Cortical Surface Thickness as a Classifier: Boosting for Autism Classification

MICCAI '08 Proceedings of the 11th international conference on Medical Image Computing and Computer-Assisted Intervention - Part I
Entropy Regularized LPBoost

ALT '08 Proceedings of the 19th international conference on Algorithmic Learning Theory
Ensemble of support vector machines for land cover classification

International Journal of Remote Sensing
Automated Ham Quality Classification Using Ensemble Unsupervised Mapping Models

KES '07 Knowledge-Based Intelligent Information and Engineering Systems and the XVII Italian Workshop on Neural Networks on Proceedings of the 11th International Conference
gBoost: a mathematical programming approach to graph classification and regression

Machine Learning
Rule learning with monotonicity constraints

ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Decision Templates Based RBF Network for Tree-Structured Multiple Classifier Fusion

MCS '09 Proceedings of the 8th International Workshop on Multiple Classifier Systems
On similarities between inference in game theory and machine learning

Journal of Artificial Intelligence Research
Exploiting known taxonomies in learning overlapping concepts

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Learning a scene background model via classification

IEEE Transactions on Signal Processing
Linear Programming Boosting by Column and Row Generation

DS '09 Proceedings of the 12th International Conference on Discovery Science
Least absolute policy iteration for robust value function approximation

ICRA'09 Proceedings of the 2009 IEEE international conference on Robotics and Automation
Representation and feature selection using multiple kernel learning

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Binarized Support Vector Machines

INFORMS Journal on Computing
A new maximum margin algorithm for one-class problems and its boosting implementation

Pattern Recognition
On the sparseness of 1-norm support vector machines

Neural Networks
Linear time maximum margin clustering

IEEE Transactions on Neural Networks
Boosting unsupervised competitive learning ensembles

ICANN'07 Proceedings of the 17th international conference on Artificial neural networks
Boosting through optimization of margin distributions

IEEE Transactions on Neural Networks
Action categorization by structural probabilistic latent semantic analysis

Computer Vision and Image Understanding
LACBoost and FisherBoost: optimally building cascade classifiers

ECCV'10 Proceedings of the 11th European conference on Computer vision: Part II
NDPMine: efficiently mining discriminative numerical features for pattern-based classification

ECML PKDD'10 Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part II
Learning with ensembles of randomized trees: new insights

ECML PKDD'10 Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part III
Sparse substring pattern set discovery using linear programming boosting

DS'10 Proceedings of the 13th international conference on Discovery science
Novel kernel-based recognizers of human actions

EURASIP Journal on Advances in Signal Processing - Special issue on video analysis for human behavior understanding
Asymmetric totally-corrective boosting for real-time object detection

ACCV'10 Proceedings of the 10th Asian conference on Computer vision - Volume Part I
Totally-corrective multi-class boosting

ACCV'10 Proceedings of the 10th Asian conference on Computer vision - Volume Part IV
Cloosting: clustering data with boosting

MCS'11 Proceedings of the 10th international conference on Multiple classifier systems
Solving semi-infinite linear programs using boosting-like methods

ALT'06 Proceedings of the 17th international conference on Algorithmic Learning Theory
Infinite ensemble learning with support vector machines

ECML'05 Proceedings of the 16th European conference on Machine Learning
Visual classification of images by learning geometric appearances through boosting

ANNPR'06 Proceedings of the Second international conference on Artificial Neural Networks in Pattern Recognition
A simple feature extraction for high dimensional image representations

SLSFS'05 Proceedings of the 2005 international conference on Subspace, Latent Structure and Feature Selection
Sparse weighted voting classifier selection and its linear programming relaxations

Information Processing Letters
Bootstrapping Boosted Random Ferns for discriminative and efficient object classification

Pattern Recognition
An Improved Branch-and-Bound Method for Maximum Monomial Agreement

INFORMS Journal on Computing
Positive semidefinite metric learning using boosting-like algorithms

The Journal of Machine Learning Research
Review: Supervised classification and mathematical optimization

Computers and Operations Research
Reduced analytical dependency modeling for classifier fusion

ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part III
A robust and efficient doubly regularized metric learning approach

ECCV'12 Proceedings of the 12th European conference on Computer Vision - Volume Part IV
Training mahalanobis kernels by linear programming

ICANN'12 Proceedings of the 22nd international conference on Artificial Neural Networks and Machine Learning - Volume Part II
Accelerated max-margin multiple kernel learning

Applied Intelligence
Online learning with multiple kernels: A review

Neural Computation
Dynamic customer lifetime value prediction using longitudinal data: An improved multiple kernel SVR approach

Knowledge-Based Systems
Particle swarm classification: A survey and positioning

Pattern Recognition
Learning with infinitely many features

Machine Learning
Fast training of effective multi-class boosting using coordinate descent optimization

ACCV'12 Proceedings of the 11th Asian conference on Computer Vision - Volume Part II
A fast algorithm for kernel 1-norm support vector machines

Knowledge-Based Systems
Analysis of programming properties and the row-column generation method for 1-norm support vector machines

Neural Networks
Fully corrective boosting with arbitrary loss and regularization

Neural Networks
A biased selection strategy for information recycling in Boosting cascade visual-object detectors

Pattern Recognition Letters

Quantified Score

Hi-index	0.00

Visualization

Abstract

We examine linear program (LP) approaches to boosting and demonstrate their efficient solution using LPBoost, a column generation based simplex method. We formulate the problem as if all possible weak hypotheses had already been generated. The labels produced by the weak hypotheses become the new feature space of the problem. The boosting task becomes to construct a learning function in the label space that minimizes misclassification error and maximizes the soft margin. We prove that for classification, minimizing the 1-norm soft margin error function directly optimizes a generalization error bound. The equivalent linear program can be efficiently solved using column generation techniques developed for large-scale optimization problems. The resulting LPBoost algorithm can be used to solve any LP boosting formulation by iteratively optimizing the dual misclassification costs in a restricted LP and dynamically generating weak hypotheses to make new LP columns. We provide algorithms for soft margin classification, confidence-rated, and regression boosting problems. Unlike gradient boosting algorithms, which may converge in the limit only, LPBoost converges in a finite number of iterations to a global solution satisfying mathematically well-defined optimality conditions. The optimal solutions of LPBoost are very sparse in contrast with gradient based methods. Computationally, LPBoost is competitive in quality and computational cost to AdaBoost.