Diagrammatic derivation of gradient algorithms for neural networks

Authors:
Eric A. Wan;Françoise Beaufays
Affiliations:
Department of Electrical Engineering and Applied Physics, Oregon Graduate Institute of Science & Technology, P.O. Box 91000, Portland, OR 97291 USA;Department of Electrical Engineering, Stanford University, Stanford, CA 94305-4055 USA
Venue:
Neural Computation
Year:
1996

Citing 10
Cited 13

Multilayer feedforward networks are universal approximators

Neural Networks
Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations
Learning a trajectory using adjoint functions and teacher forcing

Neural Networks
Neural networks and nonlinear adaptive filtering: unifying concepts and new algorithms

Neural Computation
Relating real-time backpropagation and backpropagation-through-time: an application of flow graph interreciprocity

Neural Computation
Finite impulse response neural networks with applications in time series prediction

Finite impulse response neural networks with applications in time series prediction
Optimal terminal control using feedforward neural networks

Optimal terminal control using feedforward neural networks
Fir and iir synapses, a new neural network architecture for time series modeling

Neural Computation
Backpropagation applied to handwritten zip code recognition

Neural Computation
A learning algorithm for continually running fully recurrent neural networks

Neural Computation

Gradient Computation of Continuous-Time Cellular Neural/Nonlinear Networks with Linear Templates via the CNN Universal Machine

Neural Processing Letters
On the Need for a Neural Abstract Machine

Sequence Learning - Paradigms, Algorithms, and Applications
MIGA, A Software Tool for Nonlinear System Modelling with Modular Neural Networks

Applied Intelligence
A Signal-Flow-Graph Approach to On-line Gradient Calculation

Neural Computation
Locally recurrent neural networks for wind speed prediction using spatial correlation

Information Sciences: an International Journal
Adjoint Systems for Models of Cell Signaling Pathways and their Application to Parameter Fitting

IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB)
A fuzzy-neural multi-model for nonlinear systems identification and control

Fuzzy Sets and Systems
Centralized Indirect Control of an Anaerobic Digestion Bioprocess Using Recurrent Neural Identifier

AIMSA '08 Proceedings of the 13th international conference on Artificial Intelligence: Methodology, Systems, and Applications
Direct Adaptive Soft Computing Neural Control of a Continuous Bioprocess via Second Order Learning

MICAI '09 Proceedings of the 8th Mexican International Conference on Artificial Intelligence
On-line learning algorithm based on signal flow graph theory for PID neural networks

CCDC'09 Proceedings of the 21st annual international conference on Chinese control and decision conference
Locally recurrent neural networks for long-term wind speed and power prediction

Neurocomputing
Sliding mode control of a hydrocarbon degradation in biopile system using recurrent neural network model

MICAI'07 Proceedings of the artificial intelligence 6th Mexican international conference on Advances in artificial intelligence
Recurrent neural control of a continuous bioprocess using first and second order learning

MICAI'12 Proceedings of the 11th Mexican international conference on Advances in Computational Intelligence - Volume Part II

Quantified Score

Hi-index	0.00

Visualization

Abstract

Deriving gradient algorithms for time-dependent neural network structures typically requires numerous chain rule expansions, diligent bookkeeping, and careful manipulation of terms. In this paper, we show how to derive such algorithms via a set of simple block diagram manipulation rules. The approach provides a common framework to derive popular algorithms including backpropagation and backpropagation-through-time without a single chain rule expansion. Additional examples are provided for a variety of complicated architectures to illustrate both the generality and the simplicity of the approach.