Letters: Alleviating the problem of local minima in Backpropagation through competitive learning

Authors:
Qun Dai;Ningzhong Liu
Affiliations:
Department of Computer Science and Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China;Department of Computer Science and Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China
Venue:
Neurocomputing
Year:
2012

Citing 7
Cited 0

Approximation theory and feedforward networks

Neural Networks
On the Problem of Local Minima in Backpropagation

IEEE Transactions on Pattern Analysis and Machine Intelligence
Original Contribution: An analysis of premature saturation in back propagation learning

Neural Networks
Premature saturation in backpropagation networks: mechanism and necessary conditions

Neural Networks
Three methods to speed up the training of feedforward and feedback perceptrons

Neural Networks
Avoiding local minima in feedforward neural networks by simultaneous learning

AI'07 Proceedings of the 20th Australian joint conference on Advances in artificial intelligence
Simulated annealing and weight decay in adaptive learning: the SARPROP algorithm

IEEE Transactions on Neural Networks

Quantified Score

Hi-index	0.01

Visualization

Abstract

The backpropagation (BP) algorithm is widely recognized as a powerful tool for training feedforward neural networks (FNNs). However, since the algorithm employs the steepest descent technique to adjust the network weights, it suffers from a slow convergence rate and often produces suboptimal solutions, which are the two major drawbacks of BP. This paper proposes a modified BP algorithm which can remarkably alleviate the problem of local minima confronted with by the standard BP (SBP). As one output of the modified training procedure, a bucket of all the possible solutions of weights matrices found during training is acquired, among which the best solution is chosen competitively based upon their performances on a validation dataset. Simulations are conducted on four benchmark classification tasks to compare and evaluate the classification performances and generalization capabilities of the proposed modified BP and SBP.