Stochastic complexity for mixture of exponential families in generalized variational Bayes

Authors:
Kazuho Watanabe;Sumio Watanabe
Affiliations:
Department of Computational Intelligence and Systems Science, Tokyo Institute of Technology, Japan;P&I Lab., Tokyo Institute of Technology, Japan
Venue:
Theoretical Computer Science
Year:
2007

Citing 11
Cited 1

Fundamentals of statistical exponential families: with applications in statistical decision theory

Fundamentals of statistical exponential families: with applications in statistical decision theory
Bayesian interpolation

Neural Computation
On some inequalities for the gamma and psi functions

Mathematics of Computation
Singularities in mixture models and upper bounds of stochastic complexity

Neural Networks
Online Model Selection Based on the Variational Bayes

Neural Computation
Algebraic Analysis for Nonidentifiable Learning Machines

Neural Computation
Stochastic complexities of general mixture models in variational Bayesian learning

Neural Networks
Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation

The Journal of Machine Learning Research
Inferring parameters and structure of latent variable models by variational bayes

UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence
Stochastic complexity of bayesian networks

UAI'03 Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence
Information-theoretic asymptotics of Bayes methods

IEEE Transactions on Information Theory

Learning finite Beta-Liouville mixture models via variational bayes for proportional data clustering

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence

Quantified Score

Hi-index	5.23

Visualization

Abstract

The Variational Bayesian learning, proposed as an approximation of the Bayesian learning, has provided computational tractability and good generalization performance in many applications. However, little has been done to investigate its theoretical properties. In this paper, we discuss the Variational Bayesian learning of the mixture of exponential families and derive the asymptotic form of the stochastic complexities in a generalized setting of the prior distribution. We show that the stochastic complexities become smaller than those of regular statistical models, which implies that the advantage of the Bayesian learning still remains in the Variational Bayesian learning. Stochastic complexity, which is called the marginal likelihood or the free energy, not only becomes important in addressing the model selection problem but also enables us to discuss the accuracy of the Variational Bayesian approach as an approximation of the true Bayesian learning. The main result also shows the effects of the prior distribution under the generalized setting.