Convergent activation dynamics in continuous time networks
Neural Networks
Multilayer feedforward networks are universal approximators
Neural Networks
Induction of finite-state languages using second-order recurrent networks
Neural Computation
Inductive Inference: Theory and Methods
ACM Computing Surveys (CSUR)
Introduction to Formal Language Theory
Introduction to Formal Language Theory
Finite state automata and simple recurrent networks
Neural Computation
A learning algorithm for continually running fully recurrent neural networks
Neural Computation
Architectural Bias in Recurrent Neural Networks - Fractal Analysis
ICANN '02 Proceedings of the International Conference on Artificial Neural Networks
Robust Implementaion of Finite Automata by Recurrent RBF Networks
SOFSEM '00 Proceedings of the 27th Conference on Current Trends in Theory and Practice of Informatics
Finite-State Computation in Analog Neural Networks: Steps towards Biologically Plausible Models?
Emergent Neural Computational Architectures Based on Neuroscience - Towards Neuroscience-Inspired Computing
The Underlying Formal Model of Algorithmic Lateral Inhibition in Motion Detection
IWINAC '07 Proceedings of the 2nd international work-conference on Nature Inspired Problem-Solving Methods in Knowledge Engineering: Interplay Between Natural and Artificial Computation, Part II
Extracting symbolic knowledge from recurrent neural networks---A fuzzy logic approach
Fuzzy Sets and Systems
Real-time motion detection by lateral inhibition in accumulative computation
Engineering Applications of Artificial Intelligence
Identification of finite state automata with a class of recurrent neural networks
IEEE Transactions on Neural Networks
Hybrid preference machines based on inspiration from neuroscience
Cognitive Systems Research
Hi-index | 0.00 |
We examine the correspondence between first-order recurrent neural networks and deterministic finite state automata. We begin with the problem of inducing deterministic finite state automata from finite training sets, that include both positive and negative examples, an NP-hard problem (Angluin and Smith 1983). We use a neural network architecture with two recurrent layers, which we argue can approximate any discrete-time, time-invariant dynamic system, with computation of the full gradient during learning. The networks are trained to classify strings as belonging or not belonging to the grammar. The training sets used contain only short strings, and the sets are constructed in a way that does not require a priori knowledge of the grammar. After training, the networks are tested using various test sets with strings of length up to 1000, and are often able to correctly classify all the test strings. These results are comparable to those obtained with second-order networks (Giles et al. 1992; Watrous and Kuhn 1992a; Zeng et al. 1993). We observe that the networks emulate finite state automata, confirming the results of other authors, and we use a vector quantization algorithm to extract deterministic finite state automata after training and during testing of the networks, obtaining a table listing the start state, accept states, reject states, all transitions from the states, as well as some useful statistics. We examine the correspondence between finite state automata and neural networks in detail, showing two major stages in the learning process. To this end, we use a graphics module, which graphically depicts the states of the network during the learning and testing phases. We examine the networks' performance when tested on strings much longer than those in the training set, noting a measure based on clustering that is correlated to the stability of the networks. Finally, we observe that with sufficiently long training times, neural networks can become true finite state automata, due to the attractor structure of their dynamics.