Singularity and Slow Convergence of the EM algorithm for Gaussian Mixtures

Authors:
Hyeyoung Park;Tomoko Ozeki
Affiliations:
School of Electrical Engineering and Computer Science, Kyungpook National University, Daegu, Korea 702-701;Department of Human & Information Science, Tokai University, Hiratsuka, Japan 259-1292
Venue:
Neural Processing Letters
Year:
2009

Citing 12
Cited 3

Local minima and plateaus in hierarchical structures of multilayer perceptions

Neural Networks
Adaptive natural gradient learning algorithms for various stochastic models

Neural Networks
Upper bound of the expected training error of neural network regression for a Gaussian noise sequence

Neural Networks
Algebraic geometrical methods for hierarchical learning machines

Neural Networks
On the problem in model selection of neural network regression in overrealizable scenario

Neural Computation
Algebraic Analysis for Nonidentifiable Learning Machines

Neural Computation
Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons

Neural Computation
Asymptotic Convergence Rate of the EM Algorithm for Gaussian Mixtures

Neural Computation
Singularities Affect Dynamics of Learning in Neuromanifolds

Neural Computation
Dynamics of learning near singularities in layered networks

Neural Computation
On convergence properties of the em algorithm for gaussian mixtures

Neural Computation
Asymptotic convergence properties of the EM algorithm with respect to the overlap in the mixture

Neurocomputing

Unsupervised topographic learning for spatiotemporal data mining

Advances in Artificial Intelligence - Special issue on machine learning paradigms for modeling spatial and temporal information in multimedia data mining
A multi-threshold segmentation approach based on Artificial Bee Colony optimization

Applied Intelligence
A comparison of nature inspired algorithms for multi-threshold image segmentation

Expert Systems with Applications: An International Journal

Quantified Score

Hi-index	0.00

Visualization

Abstract

Singularities in the parameter spaces of hierarchical learning machines are known to be a main cause of slow convergence of gradient descent learning. The EM algorithm, which is another learning algorithm giving a maximum likelihood estimator, is also suffering from its slow convergence, which often appears when the component overlap is large. We analyze the dynamics of the EM algorithm for Gaussian mixtures around singularities and show that there exists a slow manifold caused by a singular structure, which is closely related to the slow convergence of the EM algorithm. We also conduct numerical simulations to confirm the theoretical analysis. Through the simulations, we compare the dynamics of the EM algorithm with that of the gradient descent algorithm, and show that their slow dynamics are caused by the same singular structure, and thus they have the same behaviors around singularities.