Convergence analysis of online gradient method for BP neural networks

Authors:
Wei Wu;Jian Wang;Mingsong Cheng;Zhengxue Li
Affiliations:
School of Mathematical Sciences, Dalian University of Technology, Dalian, 116024, PR China;School of Mathematical Sciences, Dalian University of Technology, Dalian, 116024, PR China and School of Mathematics and Computational Sciences, Petroleum University of China, Dongying, 257061, PR ...;School of Mathematical Sciences, Dalian University of Technology, Dalian, 116024, PR China;School of Mathematical Sciences, Dalian University of Technology, Dalian, 116024, PR China
Venue:
Neural Networks
Year:
2011

Citing 10
Cited 9

Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to local minima

Neural Computation
Parameter convergence and learning curves for neural networks

Neural Computation
Deterministic convergence of an online gradient method for neural networks

Journal of Computational and Applied Mathematics - Selected papers of the international symposium on applied mathematics, August 2000, Dalian, China
The general inefficiency of batch training for gradient descent learning

Neural Networks
Convergence of an online gradient method for feedforward neural networks with stochastic inputs

Journal of Computational and Applied Mathematics - Special issue on proceedings of the international symposium on computational mathematics and applications
Theoretical analysis of batch and on-line training for gradient descent learning in neural networks

Neurocomputing
Boundedness and convergence of online gradient method with penalty for feedforward neural networks

IEEE Transactions on Neural Networks
When does online BP training converge?

IEEE Transactions on Neural Networks
A novel training scheme for multilayered perceptrons to realize proper generalization and incremental learning

IEEE Transactions on Neural Networks
Deterministic convergence of an online gradient method for BP neural networks

IEEE Transactions on Neural Networks

Deterministic convergence of conjugate gradient method for feedforward neural networks

Neurocomputing
Convergence of an online gradient method for BP neural networks with stochastic inputs

ICNC'05 Proceedings of the First international conference on Advances in Natural Computation - Volume Part I
Letters: A new adaptive momentum algorithm for split-complex recurrent neural networks

Neurocomputing
Computational properties and convergence analysis of BPNN for cyclic and almost cyclic learning with penalty

Neural Networks
Efficient self-organizing multilayer neural network for nonlinear system modeling

Neural Networks
Convergence of chaos injection-based batch backpropagation algorithm for feedforward neural networks

ISNN'13 Proceedings of the 10th international conference on Advances in Neural Networks - Volume Part I
Adaptive control based on IF-THEN rules for grasping force regulation with unknown contact mechanism

Robotics and Computer-Integrated Manufacturing
Monotonicity and convergence of asynchronous update gradient method for ridge polynomial neural network

Neurocomputing
Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Neurocomputing

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper considers a class of online gradient learning methods for backpropagation (BP) neural networks with a single hidden layer. We assume that in each training cycle, each sample in the training set is supplied in a stochastic order to the network exactly once. It is interesting that these stochastic learning methods can be shown to be deterministically convergent. This paper presents some weak and strong convergence results for the learning methods, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. The conditions on the activation function and the learning rate to guarantee the convergence are relaxed compared with the existing results. Our convergence results are valid for not only S-S type neural networks (both the output and hidden neurons are Sigmoid functions), but also for P-P, P-S and S-P type neural networks, where S and P represent Sigmoid and polynomial functions, respectively.