Analyzing the weight dynamics of recurrent learning algorithms

Authors:
Ulf D. Schiller;Jochen J. Steil
Affiliations:
Neuroinformatics Group, Faculty of Technology, Bielefeld University, P.O. Box 10 01 31, D-33501 Bielefield, Germany;Neuroinformatics Group, Faculty of Technology, Bielefeld University, P.O. Box 10 01 31, D-33501 Bielefield, Germany
Venue:
Neurocomputing
Year:
2005

Citing 4
Cited 7

The vanishing gradient problem during learning recurrent neural nets and problem solutions

International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems
Real-time computing without stable states: a new framework for neural computation based on perturbations

Neural Computation
New results on recurrent network training: unifying the algorithms and accelerating convergence

IEEE Transactions on Neural Networks
Gradient calculations for dynamic recurrent neural networks: a survey

IEEE Transactions on Neural Networks

2007 Special Issue: Optimization and applications of echo state networks with leaky- integrator neurons

Neural Networks
2007 Special Issue: Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning

Neural Networks
Improving reservoirs using intrinsic plasticity

Neurocomputing
Online stability of backpropagation-decorrelation recurrent learning

Neurocomputing
Memory in backpropagation-decorrelation O(N) efficient online recurrent learning

ICANN'05 Proceedings of the 15th international conference on Artificial neural networks: formal models and their applications - Volume Part II
Survey: Reservoir computing approaches to recurrent neural network training

Computer Science Review
Time series classification for the prediction of dialysis in critically ill patients using echo statenetworks

Engineering Applications of Artificial Intelligence

Quantified Score

Hi-index	0.01

Visualization

Abstract

We provide insights into the organization and dynamics of recurrent online training algorithms by comparing real time recurrent learning (RTRL) with a new continuous-time online algorithm. The latter is derived in the spirit of a recent approach introduced by Atiya and Parlos (IEEE Trans. Neural Networks 11 (3) (2000) 697), which leads to non-gradient search directions. We refer to this approach as Atiya-Parlos learning (APRL) and interpret it with respect to its strategy to minimize the standard quadratic error. Simulations show that the different approaches of RTRL and APRL lead to qualitatively different weight dynamics. A formal analysis of the one-output behavior of APRL further reveals that the weight dynamics favor a functional partition of the network into a fast output layer and a slower dynamical reservoir, whose rates of weight change are closely coupled.