A framework for the development of globally convergent adaptive learning rate algorithms

Authors:
G. D. Magoulas;V. P. Plagianakos;G. S. Androulakis;M. N. Vrahatis
Affiliations:
Department of Informatics, University of Athens, GR-157.84 Athens, GREECE and University of Patras Artificial Intelligence Research Center, GR-261.10 Patras, GREECE;Department of Mathematics, University of Patras, GR-261.10 Patras, GREECE and University of Patras Artificial Intelligence Research Center, GR-261.10 Patras, GREECE;Department of Mathematics, University of Patras, GR-261.10 Patras, GREECE and University of Patras Artificial Intelligence Research Center, GR-261.10 Patras, GREECE;Department of Mathematics, University of Patras, GR-261.10 Patras, GREECE and University of Patras Artificial Intelligence Research Center, GR-261.10 Patras, GREECE
Venue:
Progress in computer research
Year:
2001

Citing 12
Cited 0

Learning internal representations by error propagation

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1
Rescaling of variables in back propagation learning

Neural Networks
Original Contribution: A scaled conjugate gradient algorithm for fast supervised learning

Neural Networks
Original Contribution: An analysis of premature saturation in back propagation learning

Neural Networks
Effective backpropagation training with variable stepsize

Neural Networks
Optimization: algorithms and consistent approximations

Optimization: algorithms and consistent approximations
Improving the convergence of the backpropagation algorithm using learning rate adaptation methods

Neural Computation
Iterative solution of nonlinear equations in several variables

Iterative solution of nonlinear equations in several variables
A class of gradient unconstrained minimization algorithms with adaptive stepsize

Journal of Computational and Applied Mathematics
Automatic Learning Rate Maximization in Large Adaptive Machines

Advances in Neural Information Processing Systems 5, [NIPS Conference]
Acceleration Techniques for the Backpropagation Algorithm

Proceedings of the EURASIP Workshop 1990 on Neural Networks
Numerical Methods for Unconstrained Optimization and Nonlinear Equations (Classics in Applied Mathematics, 16)

Numerical Methods for Unconstrained Optimization and Nonlinear Equations (Classics in Applied Mathematics, 16)

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we propose a framework for developing globally convergent batch training algorithms with adaptive learning rate. The proposed framework provides conditions under which global convergence is guaranteed for adaptive learning rate training algorithms. To this end, the learning rate is appropriately tuned along the given descent direction. Providing conditions regarding the search direction and the corresponding stepsize length this framework can also guarantee global convergence for training algorithms that use a different learning rate for each weight. To illustrate the effectiveness of the proposed approach on various training algorithms simulation results are provided.