On the geometry of feedforward neural network error surfaces
Neural Computation
Online learning in radial basis function networks
Neural Computation
Natural gradient works efficiently in learning
Neural Computation
Algebraic geometrical methods for hierarchical learning machines
Neural Networks
Difficulty of Singularity in Population Coding
Neural Computation
Algebraic Analysis for Nonidentifiable Learning Machines
Neural Computation
Singularities Affect Dynamics of Learning in Neuromanifolds
Neural Computation
On the construction and training of reformulated radial basis function neural networks
IEEE Transactions on Neural Networks
Dynamics of Learning in Multilayer Perceptrons Near Singularities
IEEE Transactions on Neural Networks
Singularity and Slow Convergence of the EM algorithm for Gaussian Mixtures
Neural Processing Letters
| Hi-index | 0.00 |
We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the universal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depending on the stability of the singularity and the initial parameters of learning. The results presented in this letter are universally applicable to a wide class of hierarchical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate work.