Learning in fully recurrent neural networks by approaching tangent planes to constraint surfaces

Authors:
P. May;E. Zhou;C. W. Lee
Affiliations:
K College, Brook Street, Tonbridge, Kent, TN9 2PW, UK;Faculty of Advanced Engineering and Sciences, University of Bolton, Bolton, BL3 5AB, UK;Faculty of Advanced Engineering and Sciences, University of Bolton, Bolton, BL3 5AB, UK
Venue:
Neural Networks
Year:
2012

Citing 15
Cited 0

Introduction to the theory of neural computation

Introduction to the theory of neural computation
An efficient gradient-based algorithm for on-line training of recurrent network trajectories

Neural Computation
A fixed size storage O(n3) time complexity learning algorithm for fully recurrent continually running networks

Neural Computation
Learning in neural networks by using tangent planes to constraint surfaces

Neural Networks
Original Contribution: A method for improving the real-time recurrent learning algorithm

Neural Networks
Gradient-based learning algorithms for recurrent networks and their computational complexity

Backpropagation
Training feedforward neural networks: an algorithm giving improved generalization

Neural Networks
Online Symbolic-Sequence Prediction with Discrete-Time Recurrent Neural Networks

ICANN '01 Proceedings of the International Conference on Artificial Neural Networks
Identification of nonlinear discrete-time systems using raised-cosine radial basis function networks

International Journal of Systems Science
Learning Beyond Finite Memory in Recurrent Networks of Spiking Neurons

Neural Computation
Generalized locally recurrent probabilistic neural networks with application to text-independent speaker verification

Neurocomputing
Adaptive recurrent cerebellar model articulation controller for linear ultrasonic motor with optimal learning rates

Neurocomputing
A learning algorithm for continually running fully recurrent neural networks

Neural Computation
New results on recurrent network training: unifying the algorithms and accelerating convergence

IEEE Transactions on Neural Networks
Identification and control of dynamical systems using neural networks

IEEE Transactions on Neural Networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we present a new variant of the online real time recurrent learning algorithm proposed by Williams and Zipser (1989). Whilst the original algorithm utilises gradient information to guide the search towards the minimum training error, it is very slow in most applications and often gets stuck in local minima of the search space. It is also sensitive to the choice of learning rate and requires careful tuning. The new variant adjusts weights by moving to the tangent planes to constraint surfaces. It is simple to implement and requires no parameters to be set manually. Experimental results show that this new algorithm gives significantly faster convergence whilst avoiding problems like local minima.