Equations of states in singular statistical estimation

Authors:
Sumio Watanabe
Affiliations:
Precision and Intelligence Laboratory, Tokyo Institute of Technology, 4259 Nagatsuda Midori-ku, 226-8503 Yokohama Japan
Venue:
Neural Networks
Year:
2010

Citing 11
Cited 6

Algebraic geometrical methods for hierarchical learning machines

Neural Networks
On the problem in model selection of neural network regression in overrealizable scenario

Neural Computation
Learning coefficients of layered models when the true distribution mismatches the singularities

Neural Computation
On the asymptotic distribution of the least-squares estimators in unidentifiable models

Neural Computation
Singularities in mixture models and upper bounds of stochastic complexity

Neural Networks
Algebraic Analysis for Nonidentifiable Learning Machines

Neural Computation
Singularities Affect Dynamics of Learning in Neuromanifolds

Neural Computation
Stochastic complexities of reduced rank regression in Bayesian estimation

Neural Networks
Algebraic geometry and stochastic complexity of hidden Markov models

Neurocomputing
Learning efficiency of redundant neural networks in Bayesian estimation

IEEE Transactions on Neural Networks
Singularities in complete bipartite graph-type Boltzmann machines and upper bounds of stochastic complexities

IEEE Transactions on Neural Networks

Bayesian joint optimization for topic model and clustering

ICANN'10 Proceedings of the 20th international conference on Artificial neural networks: Part I
High dimensional non-linear modeling with Bayesian mixture of CCA

ICONIP'10 Proceedings of the 17th international conference on Neural information processing: theory and algorithms - Volume Part I
Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory

The Journal of Machine Learning Research
Two design methods of hyperparameters in variational Bayes learning for Bernoulli mixtures

Neurocomputing
Bayesian spectral deconvolution with the exchange Monte Carlo method

Neural Networks
Learning coefficient of generalization error in bayesian estimation and vandermonde matrix-type singularity

Neural Computation

Quantified Score

Hi-index	0.00

Visualization

Abstract

Learning machines that have hierarchical structures or hidden variables are singular statistical models because they are nonidentifiable and their Fisher information matrices are singular. In singular statistical models, neither does the Bayes a posteriori distribution converge to the normal distribution nor does the maximum likelihood estimator satisfy asymptotic normality. This is the main reason that it has been difficult to predict their generalization performance from trained states. In this paper, we study four errors, (1) the Bayes generalization error, (2) the Bayes training error, (3) the Gibbs generalization error, and (4) the Gibbs training error, and prove that there are universal mathematical relations among these errors. The formulas proved in this paper are equations of states in statistical estimation because they hold for any true distribution, any parametric model, and any a priori distribution. Also we show that the Bayes and Gibbs generalization errors can be estimated by Bayes and Gibbs training errors, and we propose widely applicable information criteria that can be applied to both regular and singular statistical models.