On the Problem of Local Minima in Backpropagation

Authors:
Marco Gori;Alberto Tesi
Affiliations:
-;-
Venue:
IEEE Transactions on Pattern Analysis and Machine Intelligence
Year:
1992

Citing 8
Cited 57

VLSI Implementation of a Neural Network Model

Computer
Perceptrons: expanded edition

Perceptrons: expanded edition
Neural networks and principal component analysis: learning from examples without local minima

Neural Networks
Multilayer feedforward networks are universal approximators

Neural Networks
Phonetically-based multi-layered neural networks for classification

Speech Communication - Neurospeech
Learning internal representations by error propagation

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1
An introduction to neural and electronic networks

An introduction to neural and electronic networks
Introduction to matrix analysis (2nd ed.)

Introduction to matrix analysis (2nd ed.)

Fuzzy pattern classification and the connectionist approach

Pattern Recognition Letters - Special issue on fuzzy set technology in pattern recognition
Are Multilayer Perceptrons Adequate for Pattern Recognition and Verification?

IEEE Transactions on Pattern Analysis and Machine Intelligence
Fast Convergent Generalized Back-Propagation Algorithm with Constant Learning Rate

Neural Processing Letters
A Fast Neural Learning Vision System for Crowd Estimation at Underground Stations Platform

Neural Processing Letters
Improving the convergence of the backpropagation algorithm using learning rate adaptation methods

Neural Computation
Pareto-optimal formulations for cost versus colorimetric accuracy trade-offs in printer color management

ACM Transactions on Graphics (TOG)
An Efficient Gradient Forecasting Search Method Utilizing the Discrete Difference Equation Prediction Model

Applied Intelligence
Computerised Auto-Scoring System Based Upon Feature Extraction and Neural Network Technologies

Journal of Intelligent and Robotic Systems
A Fast Heuristic Global Learning Algorithm for Multilayer Neural Networks

Neural Processing Letters
Unified Integration of Explicit Knowledge and Learning by Example in Recurrent Networks

IEEE Transactions on Knowledge and Data Engineering
A global optimum approach for one-layer neural networks

Neural Computation
Neural Network Learning Using Low-Discrepancy Sequence

AI '99 Proceedings of the 12th Australian Joint Conference on Artificial Intelligence: Advanced Topics in Artificial Intelligence
A Note on Learning Automata Based Schemes for Adaptation of BP Parameters

IDEAL '00 Proceedings of the Second International Conference on Intelligent Data Engineering and Automated Learning, Data Mining, Financial Engineering, and Intelligent Agents
Genetic Programming Discovers Efficient Learning Rules for the Hidden and Output Layers of Freeforward Neural Networks

Proceedings of the Second European Workshop on Genetic Programming
Efficient Learning in Adaptive Processing of Data Structures

Neural Processing Letters
Discovering efficient learning rules for feedforward neural networks using genetic programming

Recent advances in intelligent paradigms and applications
Genetic Evolution Processing of Classification

IEEE Transactions on Knowledge and Data Engineering
An effective learning of neural network by using RFBP learning algorithm

Information Sciences—Informatics and Computer Science: An International Journal
2005 Special Issue: The loading problem for recursive neural networks

Neural Networks - Special issue on neural networks and kernel methods for structured domains
Improved sign-based learning algorithm derived by the composite nonlinear Jacobi process

Journal of Computational and Applied Mathematics - Special issue: The international conference on computational methods in sciences and engineering 2004
Parameterisation and evaluation of a Bayesian network for use in an ecological risk assessment

Environmental Modelling & Software
Probabilistic based recursive model for adaptive processing of data structures

Expert Systems with Applications: An International Journal
A novel ANN-based service selection model for ubiquitous computing environments

Journal of Network and Computer Applications
A Study on Chronic Obstructive Pulmonary Disease Diagnosis Using Multilayer Neural Networks

Journal of Medical Systems
A comparative study on thyroid disease diagnosis using neural networks

Expert Systems with Applications: An International Journal
Particle swarm optimization with adaptive population size and its application

Applied Soft Computing
A novel intelligent service selection algorithm and application for ubiquitous web services environment

Expert Systems with Applications: An International Journal
A comparative study on diabetes disease diagnosis using neural networks

Expert Systems with Applications: An International Journal
Combination of modified BPNN algorithms and an efficient feature selection method for text categorization

Information Processing and Management: an International Journal
A novel soft cluster neural network for the classification of suspicious areas in digital mammograms

Pattern Recognition
On the efficient classification of data structures by neural networks

IJCAI'97 Proceedings of the Fifteenth international joint conference on Artifical intelligence - Volume 2
Computational capabilities of graph neural networks

IEEE Transactions on Neural Networks
The multi-phase method in fast learning algorithms

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
A new convex objective function for the supervised learning of single-layer neural networks

Pattern Recognition
Characterization of training errors in supervised learning using gradient-based rules

Neural Networks
Classification methods using neural networks and partial precedence algorithms for differential medical diagnosis: a case study

ICCSA'03 Proceedings of the 2003 international conference on Computational science and its applications: PartI
Avoiding local minima in feedforward neural networks by simultaneous learning

AI'07 Proceedings of the 20th Australian joint conference on Advances in artificial intelligence
Classification technology for automatic surface defects detection of steel strip based on improved BP algorithm

ICNC'09 Proceedings of the 5th international conference on Natural computation
Chest diseases diagnosis using artificial neural networks

Expert Systems with Applications: An International Journal
A reliable resilient backpropagation method with gradient ascent

ICIC'06 Proceedings of the 2006 international conference on Intelligent computing: Part II
A Study on Hepatitis Disease Diagnosis Using Multilayer Neural Network with Levenberg Marquardt Training Algorithm

Journal of Medical Systems
Indoor location sensing using geo-magnetism

MobiSys '11 Proceedings of the 9th international conference on Mobile systems, applications, and services
Theory and practice on information granule matrix

FSKD'06 Proceedings of the Third international conference on Fuzzy Systems and Knowledge Discovery
An approach based on probabilistic neural network for diagnosis of Mesothelioma's disease

Computers and Electrical Engineering
Image compression with a dynamic autoassociative neural network

Mathematical and Computer Modelling: An International Journal
Letters: Alleviating the problem of local minima in Backpropagation through competitive learning

Neurocomputing
Global hybrid ant bee colony algorithm for training artificial neural networks

ICCSA'12 Proceedings of the 12th international conference on Computational Science and Its Applications - Volume Part I
Addressing the local minima problem by output monitoring and modification algorithms

ISNN'12 Proceedings of the 9th international conference on Advances in Neural Networks - Volume Part I
Magnified gradient function to improve first-order gradient-based learning algorithms

ISNN'12 Proceedings of the 9th international conference on Advances in Neural Networks - Volume Part I
Multi-phase fast learning algorithms for solving the local minimum problem in feed-forward neural networks

ISNN'12 Proceedings of the 9th international conference on Advances in Neural Networks - Volume Part I
Back-propagation with diversive curiosity: An automatic conversion from search stagnation to exploration

Applied Soft Computing
Adaptive Resonance Theory: How a brain learns to consciously attend, learn, and recognize a changing world

Neural Networks
G-HABC Algorithm for Training Artificial Neural Networks

International Journal of Applied Metaheuristic Computing
Optimized neural network ensemble by combination of particle swarm optimization and differential evolution

ISNN'13 Proceedings of the 10th international conference on Advances in Neural Networks - Volume Part I
An Optimization Rule for In Silico Identification of Targeted Overproduction in Metabolic Pathways

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
Global Artificial Bee Colony-Levenberq-Marquardt GABC-LM Algorithm for Classification

International Journal of Applied Evolutionary Computation
Let a biogeography-based optimizer train your Multi-Layer Perceptron

Information Sciences: an International Journal

Quantified Score

Hi-index	0.15

Visualization

Abstract

The authors propose a theoretical framework for backpropagation (BP) in order to identify some of its limitations as a general learning procedure and the reasons for its success in several experiments on pattern recognition. The first important conclusion is that examples can be found in which BP gets stuck in local minima. A simple example in which BP can get stuck during gradient descent without having learned the entire training set is presented. This example guarantees the existence of a solution with null cost. Some conditions on the network architecture and the learning environment that ensure the convergence of the BP algorithm are proposed. It is proven in particular that the convergence holds if the classes are linearly separable. In this case, the experience gained in several experiments shows that multilayered neural networks (MLNs) exceed perceptrons in generalization to new examples.