Some results concerning off-training-set and IID error for the Gibbs and the Bayes optimal generalizers

Authors:
David H. Wolpert;Emanuel Knill;Tal Grossman
Affiliations:
NASA Ames Research Centre, Caelum Research MS 269-2, Moffet Field, CA 94035 USA email: dhw@ptolemy.arc.nasa.gov;CIC-3 Computer Research and Applications, MSB265, LANL, Los Alamos, NM 87545, USA email: knill@lanl.gov;Theoretical Division and CNLS, MS B213, LANL, Los Alamos, NM 87545, USA email: tal@goshawk.lanl.gov
Venue:
Statistics and Computing
Year:
1998

Citing 5
Cited 1

Occam's razor

Information Processing Letters
Calculation of the learning curve of Bayes optimal classification algorithm for learning a perceptron with noise

COLT '91 Proceedings of the fourth annual workshop on Computational learning theory
Toward efficient agnostic learning

COLT '92 Proceedings of the fifth annual workshop on Computational learning theory
Bounds on the Sample Complexity of Bayesian Learning Using Information Theory and the VC Dimension

Machine Learning - Special issue on computational learning theory
Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics)

Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics)

Interactive global illumination in dynamic participating media using selective photon tracing

Proceedings of the 21st spring conference on Computer graphics

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we analyse the average behaviour of the Bayes-optimal and Gibbs learning algorithms. We do this both for off-training-set error and conventional IID (independent identically distributed) error (for which test sets overlap with training sets). For the IID case we provide a major extension to one of the better known results. We also show that expected IID test set error is a non-increasing function of training set size for either algorithm. On the other hand, as we show, the expected off-training-set error for both learning algorithms can increase with training set size, for non-uniform sampling distributions. We characterize the relationship the sampling distribution must have with the prior for such an increase. We show in particular that for uniform sampling distributions and either algorithm, the expected off-training-set error is a non-increasing function of training set size. For uniform sampling distributions, we also characterize the priors for which the expected error of the Bayes-optimal algorithm stays constant. In addition we show that for the Bayes-optimal algorithm, expected off-training-set error can increase with training set size when the target function is fixed, but if and only if the expected error averaged over all targets decreases with training set size. Our results hold for arbitrary noise and arbitrary loss functions.