Fast and Stable Learning Utilizing Singular Regions of Multilayer Perceptron

Authors:
Seiya Satoh;Ryohei Nakano
Affiliations:
Department of Computer Science, Chubu University, Kasugai, Japan 487-8501;Department of Computer Science, Chubu University, Kasugai, Japan 487-8501
Venue:
Neural Processing Letters
Year:
2013

Citing 13
Cited 0

Neurocomputing

Neurocomputing
Original Contribution: Uniqueness of the weights for minimal feedforward nets with a given input-output map

Neural Networks
Partial BFGS update and efficient step-length calculation for three-layer neural networks

Neural Computation
Natural gradient works efficiently in learning

Neural Computation
XOR has no local minima: a case study in neural network error surface analysis

Neural Networks
Local minima and plateaus in hierarchical structures of multilayer perceptions

Neural Networks
Discovering Polynomials to Fit Multivariate Data Having Numeric and Nominal Variables

Progress in Discovery Science, Final Report of the Japanese Discovery Science Project
Pattern Classification (2nd Edition)

Pattern Classification (2nd Edition)
Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons

Neural Computation
Algebraic Geometry and Statistical Learning Theory

Algebraic Geometry and Statistical Learning Theory
Neural network tomography: Network replication from output surface geometry

Neural Networks
Implementing online natural gradient learning: problems and solutions

IEEE Transactions on Neural Networks
Dynamics of Learning in Multilayer Perceptrons Near Singularities

IEEE Transactions on Neural Networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

In the parameter space of MLP(J), multilayer perceptron with J hidden units, there exist flat areas called singular regions created by applying reducibility mappings to the optimal solution of MLP($$J-1$$). Since such singular regions cause serious stagnation of learning, a learning method to avoid singular regions has been desired. However, such avoiding does not guarantee the quality of the final solutions. This paper proposes a new learning method which does not avoid but makes good use of singular regions to stably and successively find excellent solutions commensurate with MLP(J). The proposed method worked well in our experiments using artificial and real data sets.