Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

Authors:
Amit Bhaya;Eugenius Kaszkurewicz
Affiliations:
Department of Electrical Engineering, Federal University of Rio de Janeiro, PEE/COPPE/UFRJ, P.O. Box 68504, Rio de Janeiro, RJ 21945-970, Brazil;Department of Electrical Engineering, Federal University of Rio de Janeiro, PEE/COPPE/UFRJ, P.O. Box 68504, Rio de Janeiro, RJ 21945-970, Brazil
Venue:
Neural Networks
Year:
2004

Citing 8
Cited 10

Matrix differential equations: a continuous realization process for linear algebra problems

Nonlinear Analysis: Theory, Methods & Applications
Efficient backpropagation learning using optimal learning rate and momentum

Neural Networks
Iterative methods for solving linear systems

Iterative methods for solving linear systems
On the momentum term in gradient descent learning algorithms

Neural Networks
Accelerating neural network training using weight extrapolations

Neural Networks
Control System Design

Control System Design
Stability of steepest descent with momentum for quadratic functions

IEEE Transactions on Neural Networks
Dynamic learning rate optimization of the backpropagation algorithm

IEEE Transactions on Neural Networks

Comparative Study of the CG and HBF ODEs Used in the Global Minimization of Nonconvex Functions

ICANN '09 Proceedings of the 19th International Conference on Artificial Neural Networks: Part I
Unified control Liapunov function based design of neural networks that aim at global minimization of nonconvex functions

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Convergence analysis of a back-propagation algorithm with adaptive momentum

Neurocomputing
Boundedness and convergence of online gradient method with penalty and momentum

Neurocomputing
Artificial neural network-based system for PET volume segmentation

Journal of Biomedical Imaging
Convergence of an online gradient method with inner-product penalty and adaptive momentum

Neurocomputing
Convergence of a Batch Gradient Algorithm with Adaptive Momentum for Neural Networks

Neural Processing Letters
Deterministic convergence of an online gradient method with momentum

ICIC'06 Proceedings of the 2006 international conference on Intelligent Computing - Volume Part I
Design of second order neural networks as dynamical control systems that aim to minimize nonconvex scalar functions

Neurocomputing
Computational properties of cyclic and almost-cyclic learning with momentum for feedforward neural networks

ISNN'12 Proceedings of the 9th international conference on Advances in Neural Networks - Volume Part I

Quantified Score

Hi-index	0.00

Visualization

Abstract

It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system.