On the emergence of rules in neural networks

Authors:
Stephen José Hanson;Michiro Negishi
Affiliations:
Psychology Department, Rutgers University, Newark, N.J.;Psychology Department, Rutgers University, Newark, N.J.
Venue:
Neural Computation
Year:
2002

Citing 9
Cited 6

Connectionism and cognitive architecture: a critical analysis

Connections and symbols
A stochastic version of the delta rule

CNLS '89 Proceedings of the ninth annual international conference of the Center for Nonlinear Studies on Self-organizing, Collective, and Cooperative Phenomena in Natural and Artificial Computing Networks on Emergent computation
The symbol grounding problem

CNLS '89 Proceedings of the ninth annual international conference of the Center for Nonlinear Studies on Self-organizing, Collective, and Cooperative Phenomena in Natural and Artificial Computing Networks on Emergent computation
Distributed Representations, Simple Recurrent Networks, And Grammatical Structure

Machine Learning - Connectionist approaches to language learning
Learning and extracting finite state automata with second-order recurrent neural networks

Neural Computation
Mechanisms of implicit learning: connectionist models of sequence processing

Mechanisms of implicit learning: connectionist models of sequence processing
Learning a class of large finite state machines with a recurrent neural network

Neural Networks
Discriminability-Based Transfer between Neural Networks

Advances in Neural Information Processing Systems 5, [NIPS Conference]
The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction

Neural Computation

A machine learning method for extracting symbolic knowledge from recurrent neural networks

Neural Computation
Organization of the state space of a simple recurrent network before and after training on recursive linguistic structures

Neural Networks
Free-lunch learning: modeling spontaneous recovery of memory

Neural Computation
Creating hierarchical categories using cell assemblies

Connection Science
2009 Special Issue: Discrete combinatorial circuits emerging in neural networks: A mechanism for rules of grammar in the human brain?

Neural Networks
Packing: A geometric analysis of feature selection and category formation

Cognitive Systems Research

Quantified Score

Hi-index	0.00

Visualization

Abstract

A simple associationist neural network learns to factor abstract rules (i.e., grammars) from sequences of arbitrary input symbols by inventing abstract representations that accommodate unseen symbol sets as well as unseen but similar grammars. The neural network is shown to have the ability to transfer grammatical knowledge to both new symbol vocabularies and new grammars. Analysis of the state-space shows that the network learns generalized abstract structures of the input and is not simply memorizing the input strings. These representations are context sensitive, hierarchical, and based on the state variable of the finite-state machines that the neural network has learned. Generalization to new symbol sets or grammars arises from the spatial nature of the internal representations used by the network, allowing new symbol sets to be encoded close to symbol sets that have already been learned in the hidden unit space of the network. The results are counter to the arguments that learning algorithms based on weight adaptation after each exemplar presentation (such as the long term potentiation found in the mammalian nervous system) cannot in principle extract symbolic knowledge from positive examples as prescribed by prevailing human linguistic theory and evolutionary psychology.