A gradient-based algorithm competitive with variational Bayesian EM for mixture of Gaussians

Authors:
Mikael Kuusela;Tapani Raiko;Antti Honkela;Juha Karhunen
Affiliations:
Adaptive Informatics Research Center, Helsinki University of Technology, Helsinki, Finland;Adaptive Informatics Research Center, Helsinki University of Technology, Helsinki, Finland;Adaptive Informatics Research Center, Helsinki University of Technology, Helsinki, Finland;Adaptive Informatics Research Center, Helsinki University of Technology (TKK), Helsinki, Finland
Venue:
IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Year:
2009

Citing 8
Cited 1

Original Contribution: A scaled conjugate gradient algorithm for fast supervised learning

Neural Networks
Natural gradient works efficiently in learning

Neural Computation
An unsupervised ensemble learning method for nonlinear dynamic state-space models

Neural Computation
Accelerating Cyclic Update Algorithms for Parameter Estimation by Pattern Searches

Neural Processing Letters
An Introduction to the Conjugate Gradient Method Without the Agonizing Pain

An Introduction to the Conjugate Gradient Method Without the Agonizing Pain
Online Model Selection Based on the Variational Bayes

Neural Computation
On "Natural" Learning and Pruning in Multilayered Perceptrons

Neural Computation
Pattern Recognition and Machine Learning (Information Science and Statistics)

Pattern Recognition and Machine Learning (Information Science and Statistics)

Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes

The Journal of Machine Learning Research

Quantified Score

Hi-index	0.00

Visualization

Abstract

While variational Bayesian (VB) inference is typically done with the so called VB EM algorithm, there are models where it cannot be applied because either the E-step or the M-step cannot be solved analytically. In 2007, Honkela et al. introduced a recipe for a gradient-based algorithm for VB inference that does not have such a restriction. In this paper, we derive the algorithm in the case of the mixture of Gaussians model. For the first time, the algorithm is experimentally compared to VB EM and its variant with both artificial and real data. We conclude that the algorithms are approximately as fast depending on the problem.