Analysis and Visualization of the Dynamics of Recurrent Neural Networks for Symbolic Sequences Processing

Authors:
Matej Makula;Ľubica Beňušková
Affiliations:
Faculty of Informatics and Information Technologies, Slovak Technical University, Bratislava, Slovakia 842 16;Department of Computer Science, University of Otago, Dunedin, New Zealand 9054 and Department of Applied Informatics, Faculty of Mathematics, Physics and Informatics, Comenius University, Mlynsk&# ...
Venue:
ICANN '08 Proceedings of the 18th international conference on Artificial Neural Networks, Part II
Year:
2008

Citing 7
Cited 0

Learning and extracting initial mealy automata with a modular neural network model

Neural Computation
Elements of applied bifurcation theory (2nd ed.)

Elements of applied bifurcation theory (2nd ed.)
Recurrent neural networks with small weights implement definite memory machines

Neural Computation
Architectural bias in recurrent neural networks: fractal analysis

Neural Computation
Simple Recurrent Networks Learn Context-Free and Context-Sensitive Languages by Counting

Neural Computation
Organization of the state space of a simple recurrent network before and after training on recursive linguistic structures

Neural Networks
Finite state automata and simple recurrent networks

Neural Computation

Quantified Score

Hi-index	0.00

Visualization

Abstract

Recurrent neural networks unlike feed-forward networks are able to process inputs with time context. The key role in this process is played by the dynamics of the network, which transforms input data to the recurrent layer states. Several authors have described and analyzed dynamics of small sized recurrent neural networks with two or three hidden units. In our work we introduce techniques that allow to visualize and analyze the dynamics of large recurrent neural networks with dozens units, reveal both stable and unstable points (attractors and saddle points), which are important to understand the principles of successful task processing. As a practical example of this approach, dynamics of the simple recurrent network trained by two different training algorithms on context-free language anbnwas studied.