Nonmonotonic Generalization Bias of Gaussian Mixture Models

Authors:
Shotaro Akaho;Hilbert J. Kappen
Affiliations:
Electrotechnical Laboratory, Information Science Division, Ibaraki 305-8568, Japan;RWCP Theoretical Foundation SNN, Department of Medical Physics and Biophysics, University of Nijmegen, NL 6525 EZ Nijmegen, The Netherlands
Venue:
Neural Computation
Year:
2000

Citing 5
Cited 2

A competitive modular connectionist architecture

NIPS-3 Proceedings of the 1990 conference on Advances in neural information processing systems 3
A universal theorem on learning curves

Neural Networks
Hierarchical mixtures of experts and the EM algorithm

Neural Computation
Deterministic learning rules for Boltzmann machines

Neural Networks
Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics)

Estimation of Dependences Based on Empirical Data: Springer Series in Statistics (Springer Series in Statistics)

Singularities Affect Dynamics of Learning in Neuromanifolds

Neural Computation
Stochastic complexities of general mixture models in variational Bayesian learning

Neural Networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

Theories of learning and generalization hold that the generalization bias, defined as the difference between the training error and the generalization error, increases on average with the number of adaptive parameters. This article, however, shows that this general tendency is violated for a gaussian mixture model. For temperatures just below the first symmetry breaking point, the effective number of adaptive parameters increases and the generalization bias decreases. We compute the dependence of the neural information criterion on temperature around the symmetry breaking. Our results are confirmed by numerical cross-validation experiments.