Gradient calculations for dynamic recurrent neural networks: a survey

Authors:
B. A. Pearlmutter
Affiliations:
Learning Syst. Dept., Siemens Corp. Res. Inc., Princeton, NJ
Venue:
IEEE Transactions on Neural Networks
Year:
1995

Citing 0
Cited 74

Trajectory Control of Convergent Networks

Neural Processing Letters
Evolution and development of a central pattern generator for the swimming of a Lamprey

Artificial Life
A Self-Organizing Context-Based Approach to the Tracking of Multiple Robot Trajectories

Applied Intelligence
A Stochastic Neural Model for Fast Identification of Spatiotemporal Sequences

Neural Processing Letters
A Monte Carlo EM Approach for partially observable diffusion processes: theory and applications to neural networks

Neural Computation
Real-time computing without stable states: a new framework for neural computation based on perturbations

Neural Computation
Computational intelligence based approach for the joint trajectory generation of cooperative robotic systems

Systems Analysis Modelling Simulation
Finite-state computation in analog neural networks: steps towards biologically plausible models?

Emergent neural computational architectures based on neuroscience
Learning the Dynamic Neural Networks with the Improvement of Generalization Capabilities

ICANN '02 Proceedings of the International Conference on Artificial Neural Networks
Improving Long-Term Online Prediction with Decoupled Extended Kalman Filters

ICANN '02 Proceedings of the International Conference on Artificial Neural Networks
Multi-agent FX-Market Modeling Based on Cognitive Systems

ICANN '01 Proceedings of the International Conference on Artificial Neural Networks
Modeling of the German Yield Curve by Error Correction Neural Networks

IDEAL '00 Proceedings of the Second International Conference on Intelligent Data Engineering and Automated Learning, Data Mining, Financial Engineering, and Intelligent Agents
Finite-State Computation in Analog Neural Networks: Steps towards Biologically Plausible Models?

Emergent Neural Computational Architectures Based on Neuroscience - Towards Neuroscience-Inspired Computing
Imitation: a means to enhance learning of a synthetic protolanguage in autonomous robots

Imitation in animals and artifacts
Mining heterogeneous gene expression data with time lagged recurrent neural networks

Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
On unique representations of certain dynamical systems produced by continuous-time recurrent neural networks

Neural Computation
Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets

Neural Networks
A taxonomy for spatiotemporal connectionist networks revisited: the unsupervised case

Neural Computation
Local maximum ozone concentration prediction using soft computing methodologies

Systems Analysis Modelling Simulation
Learning precise timing with lstm recurrent networks

The Journal of Machine Learning Research
Incremental training of first order recurrent neural networks to predict a context-sensitive language

Neural Networks
Evolutionary Adaptation of Nonlinear Dynamical Systems in Computational Neuroscience

Genetic Programming and Evolvable Machines
Improving generalization capabilities of dynamic neural networks

Neural Computation
Spatiotemporal Connectionist Networks: A Taxonomy and Review

Neural Computation
A Signal-Flow-Graph Approach to On-line Gradient Calculation

Neural Computation
Learning to Forget: Continual Prediction with LSTM

Neural Computation
Imitation learning with spiking neural networks and real-world devices

Engineering Applications of Artificial Intelligence
Movement prediction from real-world images using a liquid state machine

Applied Intelligence
Recurrent cerebellar loops simplify adaptive control of redundant and nonlinear motor systems

Neural Computation
Long Short-Term Memory

Neural Computation
Training Recurrent Networks by Evolino

Neural Computation
2007 Special Issue: Optimization and applications of echo state networks with leaky- integrator neurons

Neural Networks
Reinforcement learning for a biped robot based on a CPG-actor-critic method

Neural Networks
Recurrent fuzzy network design using hybrid evolutionary learning algorithms

Neurocomputing
2008 Special Issue: Central pattern generators for locomotion control in animals and robots: A review

Neural Networks
Learning long-term dependencies with recurrent neural networks

Neurocomputing
Long-term time series prediction with the NARX network: An empirical evaluation

Neurocomputing
Using acceleration measurements for activity recognition: An effective learning algorithm for constructing neural classifiers

Pattern Recognition Letters
Robust delay-distribution-dependent stability of discrete-time stochastic neural networks with time-varying delay

Neurocomputing
Identification using ANFIS with intelligent hybrid stable learning algorithm approaches and stability analysis of training methods

Applied Soft Computing
Training ANFIS as an identifier with intelligent hybrid stable learning algorithm based on particle swarm optimization and extended Kalman filter

Fuzzy Sets and Systems
Time lagged recurrent neural network for temporal gene expression classification

International Journal of Computational Intelligence in Bioinformatics and Systems Biology
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes

Anticipatory Behavior in Adaptive Learning Systems
A new hypothesis for sleep: Tuning for criticality

Neural Computation
Recurrent neuro-fuzzy system for fault detection and isolation in nuclear reactors

Advanced Engineering Informatics
A robust extended Elman backpropagation algorithm

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Temporal semantics: an adaptive resonance theory approach

IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
A novel evolutionary connectionist text summarizer (ECTS)

ASID'09 Proceedings of the 3rd international conference on Anti-Counterfeiting, security, and identification in communication
Analyzing the weight dynamics of recurrent learning algorithms

Neurocomputing
Online stability of backpropagation-decorrelation recurrent learning

Neurocomputing
Improving optimality of neural rewards regression for data-efficient batch near-optimal policy identification

ICANN'07 Proceedings of the 17th international conference on Artificial neural networks
Adaptive self-scaling non-monotone BFGS training algorithm for recurrent neural networks

ICANN'07 Proceedings of the 17th international conference on Artificial neural networks
Unsupervised learning of echo state networks: a case study in artificial embryogeny

EA'07 Proceedings of the Evolution artificielle, 8th international conference on Artificial evolution
2006: celebrating 75 years of AI - history and outlook: the next 25 years

50 years of artificial intelligence
Channel equalization using neural networks: a review

IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews
On the weight convergence of Elman networks

IEEE Transactions on Neural Networks
Constructive training of recurrent neural networks using hybrid optimization

Neurocomputing
The Bogdanov-Takens bifurcation analysis on a three dimensional recurrent neural network

Neurocomputing
Permanent oscillations in a 3-node recurrent neural network model

Neurocomputing
Identification of Duffing's equation with dynamic recurrent neural network

ISNN'05 Proceedings of the Second international conference on Advances in neural networks - Volume Part II
A robotic scenario for programmable fixed-weight neural networks exhibiting multiple behaviors

ICANNGA'11 Proceedings of the 10th international conference on Adaptive and natural computing algorithms - Volume Part I
Simulation and verification of zhang neural networks and gradient neural networks for time-varying Stein equation solving

ISNN'11 Proceedings of the 8th international conference on Advances in neural networks - Volume Part I
Recurrent kernel machines: Computing with infinite echo state networks

Neural Computation
Fast color-based object recognition independent of position and orientation

ICANN'05 Proceedings of the 15th international conference on Artificial Neural Networks: biological Inspirations - Volume Part I
Review: Situation identification techniques in pervasive computing: A review

Pervasive and Mobile Computing
Time window width influence on dynamic BPTT(h) learning algorithm performances: experimental study

ICANN'06 Proceedings of the 16th international conference on Artificial Neural Networks - Volume Part I
A nonparametric Bayesian approach toward robot learning by demonstration

Robotics and Autonomous Systems
Letters: Zhang neural network versus gradient-based neural network for time-varying linear matrix equation solving

Neurocomputing
New automated learning CPG for rhythmic patterns

Intelligent Service Robotics
Unsupervised formation of vocalization-sensitive neurons: A cortical model based on short-term and homeostatic plasticity

Neural Computation
A context layered locally recurrent neural network for dynamic system identification

Engineering Applications of Artificial Intelligence
Optimal design of neuro-mechanical oscillators

Computers and Structures
A biological gradient descent for prediction through a combination of stdp and homeostatic plasticity

Neural Computation
A linear recurrent kernel online learning algorithm with sparse updates

Neural Networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

Surveys learning algorithms for recurrent neural networks with hidden units and puts the various techniques into a common framework. The authors discuss fixed point learning algorithms, namely recurrent backpropagation and deterministic Boltzmann machines, and nonfixed point algorithms, namely backpropagation through time, Elman's history cutoff, and Jordan's output feedback architecture. Forward propagation, an on-line technique that uses adjoint equations, and variations thereof, are also discussed. In many cases, the unified presentation leads to generalizations of various sorts. The author discusses advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones continues with some “tricks of the trade” for training, using, and simulating continuous time and recurrent neural networks. The author presents some simulations, and at the end, addresses issues of computational complexity and learning speed