PAC-bayes bounds with data dependent priors

Authors:
Emilio Parrado-Hernández;Amiran Ambroladze;John Shawe-Taylor;Shiliang Sun
Affiliations:
Department of Signal Processing and Communications, University Carlos III of Madrid, Leganés, Spain;Department of Mathematics and Computer Science, Tbilisi Free University, Tbilisi, Georgia;Department of Computer Science, University College London, London, UK;Department of Computer Science and Technology, East China Normal University, Shanghai, China
Venue:
The Journal of Machine Learning Research
Year:
2012

Citing 10
Cited 1

A training algorithm for optimal margin classifiers

COLT '92 Proceedings of the fifth annual workshop on Computational learning theory
PAC-Bayesian model averaging

COLT '99 Proceedings of the twelfth annual conference on Computational learning theory
An introduction to support Vector Machines: and other kernel-based learning methods

An introduction to support Vector Machines: and other kernel-based learning methods
Covering number bounds of certain regularized linear function classes

The Journal of Machine Learning Research
Pac-bayesian generalisation error bounds for gaussian process classification

The Journal of Machine Learning Research
Rademacher and gaussian complexities: risk bounds and structural results

The Journal of Machine Learning Research
Kernel Methods for Pattern Analysis

Kernel Methods for Pattern Analysis
Tutorial on Practical Prediction Theory for Classification

The Journal of Machine Learning Research
PAC-Bayesian learning of linear classifiers

ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning
Structural risk minimization over data-dependent hierarchies

IEEE Transactions on Information Theory

A novel approach to hyperspectral band selection based on spectral shape similarity analysis and fast branch and bound search

Engineering Applications of Artificial Intelligence

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents the prior PAC-Bayes bound and explores its capabilities as a tool to provide tight predictions of SVMs' generalization. The computation of the bound involves estimating a prior of the distribution of classifiers from the available data, and then manipulating this prior in the usual PAC-Bayes generalization bound. We explore two alternatives: to learn the prior from a separate data set, or to consider an expectation prior that does not need this separate data set. The prior PAC-Bayes bound motivates two SVM-like classification algorithms, prior SVM and ν-prior SVM, whose regularization term pushes towards the minimization of the prior PAC-Bayes bound. The experimental work illustrates that the new bounds can be significantly tighter than the original PAC-Bayes bound when applied to SVMs, and among them the combination of the prior PAC-Bayes bound and the prior SVM algorithm gives the tightest bound.