Statistical learning by natural gradient descent

Authors:
H. H. Yang;S. Amari
Affiliations:
Oregon Graduate Institute of Science and Technology, Beaverton;RIKEN Brain Science Institute, Saitama, Japan
Venue:
New learning paradigms in soft computing
Year:
2002

Citing 3
Cited 0

Natural gradient works efficiently in learning

Neural Computation
Complexity issues in natural gradient descent method for training multilayer perceptrons

Neural Computation
Equivariant adaptive source separation

IEEE Transactions on Signal Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Based on stochastic perceptron models and statistical inference, we train single-layer and two-layer perceptrons by natural gradient descent. We have discovered an efficient scheme to present the Fisher information matrix of a stochastic two-layer perceptron. Based on this scheme, we have designed an algorithm to compute the natural gradient. When the input dimension n is much larger than the number of hidden neurons, the complexity of this algorithm is of order O (n). It is confirmed by simulations that the natural gradient descent learning rule is not only efficient but also robust.