Deterministic convergence of conjugate gradient method for feedforward neural networks

Authors:
Jian Wang;Wei Wu;Jacek M. Zurada
Affiliations:
School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, PR China and Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY 40292, USA ...;School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, PR China;Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY 40292, USA
Venue:
Neurocomputing
Year:
2011

Citing 17
Cited 3

Generalized Polak-Ribie`re algorithm

Journal of Optimization Theory and Applications
Diffusion approximations for the constant learning rate backpropagation algorithm and resistance to local minima

Neural Computation
Neural network design

Neural network design
A globally convergent version of the Polak-Ribière conjugate gradient method

Mathematical Programming: Series A and B
The general inefficiency of batch training for gradient descent learning

Neural Networks
Convergence of an online gradient method for feedforward neural networks with stochastic inputs

Journal of Computational and Applied Mathematics - Special issue on proceedings of the international symposium on computational mathematics and applications
Natural conjugate gradient training of multilayer perceptrons

Neurocomputing
Boundedness and convergence of online gradient method with penalty for feedforward neural networks

IEEE Transactions on Neural Networks
When does online BP training converge?

IEEE Transactions on Neural Networks
A novel recurrent neural network with one neuron and finite-time convergence for k-winners-take-all operation

IEEE Transactions on Neural Networks
Convergence analysis of three classes of split-complex gradient algorithms for complex-valued recurrent neural networks

Neural Computation
A novel recurrent neural network with finite-time convergence for linear programming

Neural Computation
Convergence analysis of online gradient method for BP neural networks

Neural Networks
Optimization method based extreme learning machine for classification

Neurocomputing
A novel training scheme for multilayered perceptrons to realize proper generalization and incremental learning

IEEE Transactions on Neural Networks
Deterministic convergence of an online gradient method for BP neural networks

IEEE Transactions on Neural Networks
Convergence of Cyclic and Almost-Cyclic Learning With Momentum for Feedforward Neural Networks

IEEE Transactions on Neural Networks

Boundedness and convergence of batch back-propagation algorithm with penalty for feedforward neural networks

Neurocomputing
Convergence of chaos injection-based batch backpropagation algorithm for feedforward neural networks

ISNN'13 Proceedings of the 10th international conference on Advances in Neural Networks - Volume Part I
Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Neurocomputing

Quantified Score

Hi-index	0.01

Visualization

Abstract

Conjugate gradient methods have many advantages in real numerical experiments, such as fast convergence and low memory requirements. This paper considers a class of conjugate gradient learning methods for backpropagation neural networks with three layers. We propose a new learning algorithm for almost cyclic learning of neural networks based on PRP conjugate gradient method. We then establish the deterministic convergence properties for three different learning modes, i.e., batch mode, cyclic and almost cyclic learning. The two deterministic convergence properties are weak and strong convergence that indicate that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. It is shown that the deterministic convergence results are based on different learning modes and dependent on different selection strategies of learning rate. Illustrative numerical examples are given to support the theoretical analysis.