The interchangeability of learning rate and gain in backpropagation neural networks

Authors:
Georg Thimm;Perry Moerland;Emile Fiesler
Affiliations:
IDIAP, CH-1920 Martigny, Switzerland;IDIAP, CH-1920 Martigny, Switzerland;IDIAP, CH-1920 Martigny, Switzerland
Venue:
Neural Computation
Year:
1996

Citing 6
Cited 6

Analysis of neural networks with redundancy

Neural Computation
Learning internal representations by error propagation

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1
Introduction to artificial neural systems

Introduction to artificial neural systems
Speed up learning and network optimization with extended back propagation

Neural Networks
Neurofuzzy adaptive modelling and control

Neurofuzzy adaptive modelling and control
Learning and generalization in neural networks

Learning and generalization in neural networks

Relating the slope of the activation function and the learning rate within a recurrent neural network

Neural Computation
On different facets of regularization theory

Neural Computation
Automatic Generation of Initial Weights and Estimation of Hidden Units for Pattern Classification Using Neural Networks

ICPR '98 Proceedings of the 14th International Conference on Pattern Recognition-Volume 2 - Volume 2
Relationships Between the A Priori and A Posteriori Errors in Nonlinear Adaptive Neural Filters

Neural Computation
A homomorphic neural network for modeling and prediction

Neural Computation
Nonlinear spline adaptive filtering

Signal Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

The backpropagation algorithm is widely used for training multilayer neural networks. In this publication the gain of its activation function(s) is investigated. In specific, it is proven that changing the gain of the activation function is equivalent to changing the learning rate and the weights. This simplifies the backpropagation learning rule by eliminating one of its parameters. The theorem can be extended to hold for some well-known variations on the backpropagation algorithm, such as using a momentum term, flat spot elimination, or adaptive gain. Furthermore, it is successfully applied to compensate for the nonstandard gain of optical sigmoids for optical neural networks.