Some greedy learning algorithms for sparse regression and classification with mercer kernels

Authors:
Prasanth B. Nair;Arindam Choudhury;Andy J. Keane
Affiliations:
Computational Engineering and Design Center, School of Engineering Sciences, University of Southampton, Highfield, Southampton SO17 1BJ, United Kingdom;Computational Engineering and Design Center, School of Engineering Sciences, University of Southampton, Highfield, Southampton SO17 1BJ, United Kingdom;Computational Engineering and Design Center, School of Engineering Sciences, University of Southampton, Highfield, Southampton SO17 1BJ, United Kingdom
Venue:
The Journal of Machine Learning Research
Year:
2003

Citing 18
Cited 13

Algorithm 686: FORTRAN subroutines for updating the QR decomposition

ACM Transactions on Mathematical Software (TOMS)
On selecting models for nonlinear time series

Physica D
Sparse Approximate Solutions to Linear Systems

SIAM Journal on Computing
The nature of statistical learning theory

The nature of statistical learning theory
Parallel Preconditioning with Sparse Approximate Inverses

SIAM Journal on Scientific Computing
Approximate Inverse Techniques for Block-Partitioned Matrices

SIAM Journal on Scientific Computing
Approximate Inverse Preconditioners via Sparse-Sparse Iterations

SIAM Journal on Scientific Computing
An equivalence between sparse approximation and support vector machines

Neural Computation
Atomic Decomposition by Basis Pursuit

SIAM Journal on Scientific Computing
On Learning Functions from Noise-Free and Noisy Samples via Occam's Razor

SIAM Journal on Computing
On the Optimality of the Backward Greedy Algorithm for the Subset Selection Problem

SIAM Journal on Matrix Analysis and Applications
Pattern Recognition and Neural Networks

Pattern Recognition and Neural Networks
Kernel Matching Pursuit

Machine Learning
Some Sparse Approximation Bounds for Regression Problems

ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Sparse Greedy Matrix Approximation for Machine Learning

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Sparse bayesian learning and the relevance vector machine

The Journal of Machine Learning Research
New Support Vector Algorithms

Neural Computation
Comparing support vector machines with Gaussian kernels to radialbasis function classifiers

IEEE Transactions on Signal Processing

Efficient computations via scalable sparse kernel partial least squares and boosted latent features

Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining
Building Sparse Large Margin Classifiers

ICML '05 Proceedings of the 22nd international conference on Machine learning
Learning inexpensive parametric design models using an augmented genetic programming technique

Artificial Intelligence for Engineering Design, Analysis and Manufacturing
Constructing a speculative kernel machine for pattern classification

Neural Networks
Kernel least-squares models using updates of the pseudoinverse

Neural Computation
A Direct Method for Building Sparse Kernel Learning Algorithms

The Journal of Machine Learning Research
A sparsity driven kernel machine based on minimizing a generalization error bound

Pattern Recognition
Updates for nonlinear discriminants

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Sparse approximation through boosting for learning large scale kernel machines

IEEE Transactions on Neural Networks
Example-dependent basis vector selection for kernel-based classifiers

ECML PKDD'10 Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part III
Building a sparse kernel classifier on riemannian manifold

VSMM'06 Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems
The support vector tree

Algorithms and Applications
Pruning least objective contribution in KMSE

Neurocomputing

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present greedy learning algorithms for building sparse nonlinear regression and classification models from observational data using Mercer kernels. Our objective is to develop efficient numerical schemes for reducing the training and runtime complexities of kernel-based algorithms applied to large datasets. In the spirit of Natarajan's greedy algorithm (Natarajan, 1995), we iteratively minimize the L2 loss function subject to a specified constraint on the degree of sparsity required of the final model or till a specified stopping criterion is reached. We discuss various greedy criteria for basis selection and numerical schemes for improving the robustness and computational efficiency. Subsequently, algorithms based on residual minimization and thin QR factorization are presented for constructing sparse regression and classification models. During the course of the incremental model construction, the algorithms are terminated using model selection principles such as the minimum descriptive length (MDL) and Akaike's information criterion (AIC). Finally, experimental results on benchmark data are presented to demonstrate the competitiveness of the algorithms developed in this paper.