A model for learning to segment temporal sequences, utilizing a mixture of RNN experts together with adaptive variance

Authors:
Jun Namikawa;Jun Tani
Affiliations:
Brain Science Institute, RIKEN, 2-1 Hirosawa, Wako-shi, Saitama, 351-0198, Japan;Brain Science Institute, RIKEN, 2-1 Hirosawa, Wako-shi, Saitama, 351-0198, Japan
Venue:
Neural Networks
Year:
2008

Citing 16
Cited 2

Learning internal representations by error propagation

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1
The Induction of Dynamical Recognizers

Machine Learning - Connectionist approaches to language learning
Hierarchical mixtures of experts and the EM algorithm

Neural Computation
Multiple paired forward and inverse models for motor control

Neural Networks - Special issue on neural control and robotics: biology and technology
Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems

Neural Networks - Special issue on organisation of computation in brain-like systems
Functional dynamics. I: articulation process

Physica D
Functional dynamics II: syntactic structure

Physica D
Learning nonregular languages: a comparison of simple recurrent networks and LSTM

Neural Computation
Recurrent neural networks with small weights implement definite memory machines

Neural Computation
Long Short-Term Memory

Neural Computation
Adaptive mixtures of local experts

Neural Computation
A learning algorithm for continually running fully recurrent neural networks

Neural Computation
Incremental learning of sequence patterns with a modular network model

Neurocomputing
Learning to imitate stochastic time series in a compositional way by chaos

Neural Networks
Codevelopmental Learning Between Human and Humanoid Robot Using a Dynamic Neural-Network Model

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics
Learning long-term dependencies with gradient descent is difficult

IEEE Transactions on Neural Networks

Learning to imitate stochastic time series in a compositional way by chaos

Neural Networks
Imitating others by composition of primitive actions: A neuro-dynamic model

Robotics and Autonomous Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper proposes a novel learning method for a mixture of recurrent neural network (RNN) experts model, which can acquire the ability to generate desired sequences by dynamically switching between experts. Our method is based on maximum likelihood estimation, using a gradient descent algorithm. This approach is similar to that used in conventional methods; however, we modify the likelihood function by adding a mechanism to alter the variance for each expert. The proposed method is demonstrated to successfully learn Markov chain switching among a set of 9 Lissajous curves, for which the conventional method fails. The learning performance, analyzed in terms of the generalization capability, of the proposed method is also shown to be superior to that of the conventional method. With the addition of a gating network, the proposed method is successfully applied to the learning of sensory-motor flows for a small humanoid robot as a realistic problem of time series prediction and generation.