Combining self-organizing maps with mixtures of experts: application to an actor-critic model of reinforcement learning in the basal ganglia

Authors:
Mehdi Khamassi;Louis-Emmanuel Martinet;Agnè/s Guillot
Affiliations:
AnimatLab –/ LIP6, F-75005 Paris, France / CNRS, UMR7606, Université/ Pierre et Marie Curie –/ Paris 6, UMR7606, Paris, France;AnimatLab –/ LIP6, F-75005 Paris, France / CNRS, UMR7606, Université/ Pierre et Marie Curie –/ Paris 6, UMR7606, Paris, France;AnimatLab –/ LIP6, F-75005 Paris, France / CNRS, UMR7606, Université/ Pierre et Marie Curie –/ Paris 6, UMR7606, Paris, France
Venue:
SAB'06 Proceedings of the 9th international conference on From Animals to Animats: simulation of Adaptive Behavior
Year:
2006

Citing 10
Cited 2

Neural networks and the bias/variance dilemma

Neural Computation
Layered control architectures in robots and vertebrates

Adaptive Behavior
Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems

Neural Networks - Special issue on organisation of computation in brain-like systems
Self-Organizing Maps

Self-Organizing Maps
Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Multiple model-based reinforcement learning

Neural Computation
A self-organising network that grows when required

Neural Networks - New developments in self-organizing maps
Applications of the self-organising map to reinforcement learning

Neural Networks - New developments in self-organizing maps
Actor-Critic Models of Reinforcement Learning in the Basal Ganglia: From Natural to Artificial Rats

Adaptive Behavior - Animals, Animats, Software Agents, Robots, Adaptive Systems
A modular neural-network model of the basal ganglia's role in learning and selecting motor behaviours

Cognitive Systems Research

Integration of an Omnidirectional Visual System with the Control Architecture of Psikharpax

SAB '08 Proceedings of the 10th international conference on Simulation of Adaptive Behavior: From Animals to Animats
A computational model of integration between reinforcement learning and task monitoring in the prefrontal cortex

SAB'10 Proceedings of the 11th international conference on Simulation of adaptive behavior: from animals to animats

Quantified Score

Hi-index	0.00

Visualization

Abstract

In a reward-seeking task performed in a continuous environment, our previous work compared several Actor-Critic (AC) architectures implementing dopamine-like reinforcement learning mechanisms in the rat's basal ganglia The task complexity imposes the coordination of several AC submodules, each module being an expert trained in a particular subset of the task We showed that the classical method where the choice of the expert to train at a given time depends on each expert's performance suffered from strong limitations We rather proposed to cluster the continuous state space by an ad hoc method that lacked autonomy and generalization abilities In the present work we have combined the mixture of experts with self-organizing maps in order to cluster autonomously the experts' responsibility space On the one hand, we find that classical Kohonen maps give very variable results: some task decompositions provide very good and stable reinforcement learning performances, whereas some others are unadapted to the task Moreover, they require the number of experts to be set a priori On the other hand, algorithms like Growing Neural Gas or Growing When Required have the property to choose autonomously and incrementally the number of experts to train They lead to good performances, even if they are still weaker than our hand-tuned task decomposition and than the best Kohonen maps that we got We finally discuss on propositions about what information to add to these algorithms, such as knowledge of current behavior, in order to make the task decomposition appropriate to the reinforcement learning process.