Constructive learning of recurrent neural networks: limitations of recurrent cascade correlation and a simple solution

Authors:
C. L. Giles;Dong Chen;Guo-Zheng Sun;Hsing-Hen Chen;Yee-Chung Lee;M. W. Goudreau
Affiliations:
Inst. for Adv. Comput. Studies, Maryland Univ., College Park, MD;-;-;-;-;-
Venue:
IEEE Transactions on Neural Networks
Year:
1995

Citing 0
Cited 16

CHILD: A First Step Towards Continual Learning

Machine Learning - Special issue on inductive transfer
Constructive Backpropagation for Recurrent Networks

Neural Processing Letters
Learning the Dynamic Neural Networks with the Improvement of Generalization Capabilities

ICANN '02 Proceedings of the International Conference on Artificial Neural Networks
On the Need for a Neural Abstract Machine

Sequence Learning - Paradigms, Algorithms, and Applications
Rules Extraction by Constructive Learning of Neural Networks and Hidden-Unit Clustering

DS '99 Proceedings of the Second International Conference on Discovery Science
Improving generalization capabilities of dynamic neural networks

Neural Computation
Universal Approximation Capability of Cascade Correlation for Structures

Neural Computation
Spatiotemporal Connectionist Networks: A Taxonomy and Review

Neural Computation
An empirical evaluation of constructive neural network algorithms in classification tasks

International Journal of Innovative Computing and Applications
A procedure to enhance identification of recurrent neural networks for simulating air-fuel ratio dynamics in SI engines

Engineering Applications of Artificial Intelligence
Topology optimization and training of recurrent neural networks with Pareto-based multi-objective algorithms: a experimental study

IWANN'07 Proceedings of the 9th international work conference on Artificial neural networks
Artificial neural networks capable of learning spatiotemporal chemical diffusion in the cortical brain

Pattern Recognition
Constructive training of recurrent neural networks using hybrid optimization

Neurocomputing
Representation and identification method of finite state automata by recurrent high-order neural networks

ICANN'05 Proceedings of the 15th international conference on Artificial neural networks: formal models and their applications - Volume Part II
Rules extraction from constructively trained neural networks based on genetic algorithms

Neurocomputing
Ontology alignment using artificial neural network for large-scale ontologies

International Journal of Metadata, Semantics and Ontologies

Quantified Score

Hi-index	0.00

Visualization

Abstract

It is often difficult to predict the optimal neural network size for a particular application. Constructive or destructive methods that add or subtract neurons, layers, connections, etc. might offer a solution to this problem. We prove that one method, recurrent cascade correlation, due to its topology, has fundamental limitations in representation and thus in its learning capabilities. It cannot represent with monotone (i.e., sigmoid) and hard-threshold activation functions certain finite state automata. We give a “preliminary” approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully-recurrent structure. We illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn