Neural Computation
Keeping the neural networks simple by minimizing the description length of the weights
COLT '93 Proceedings of the sixth annual conference on Computational learning theory
Bayesian parameter estimation via variational methods
Statistics and Computing
UAI '04 Proceedings of the 20th conference on Uncertainty in artificial intelligence
Online Model Selection Based on the Variational Bayes
Neural Computation
Algebraic Analysis for Nonidentifiable Learning Machines
Neural Computation
Singularities Affect Dynamics of Learning in Neuromanifolds
Neural Computation
Generalization Performance of Subspace Bayes Approach in Linear Neural Networks
IEICE - Transactions on Information and Systems
Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation
The Journal of Machine Learning Research
Generalization error of linear neural networks in an empirical bayes approach
IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Inferring parameters and structure of latent variable models by variational bayes
UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence
Asymptotic model selection for naive Bayesian networks
UAI'02 Proceedings of the Eighteenth conference on Uncertainty in artificial intelligence
Stochastic complexity of bayesian networks
UAI'03 Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence
Learning in linear neural networks: a survey
IEEE Transactions on Neural Networks
Analysis of Variational Bayesian Matrix Factorization
PAKDD '09 Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining
Generalization error of automatic relevance determination
ICANN'07 Proceedings of the 17th international conference on Artificial neural networks
Bayesian inference based on stationary fokker-planck sampling
Neural Computation
Theoretical Analysis of Bayesian Matrix Factorization
The Journal of Machine Learning Research
| Hi-index | 0.00 |
It is well known that in unidentifiable models, the Bayes estimation provides much better generalization performance than the maximum likelihood (ML) estimation. However, its accurate approximation by Markov chain Monte Carlo methods requires huge computational costs. As an alternative, a tractable approximation method, called the variational Bayes (VB) approach, has recently been proposed and has been attracting attention. Its advantage over the expectation maximization (EM) algorithm, often used for realizing the ML estimation, has been experimentally shown in many applications; nevertheless, it has not yet been theoretically shown. In this letter, through analysis of the simplest unidentifiable models, we theoretically show some properties of the VB approach. We first prove that in three-layer linear neural networks, the VB approach is asymptotically equivalent to a positive-part James-Stein type shrinkage estimation. Then we theoretically clarify its free energy, generalization error, and training error. Comparing them with those of the ML estimation and the Bayes estimation, we discuss the advantage of the VB approach. We also show that unlike in the Bayes estimation, the free energy and the generalization error are less simply related with each other and that in typical cases, the VB free energy well approximates the Bayes one, while the VB generalization error significantly differs from the Bayes one.