Model-based reinforcement learning: a computational model and an fMRI study

Authors:
Wako Yoshida;Shin Ishii
Affiliations:
Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 630-0192, Japan and CREST, Japan Science and Technology Agency, Japan;Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara 630-0192, Japan and CREST, Japan Science and Technology Agency, Japan
Venue:
Neurocomputing
Year:
2005

Citing 8
Cited 1

Integrated architecture for learning, planning, and reacting based on approximating dynamic programming

Proceedings of the seventh international conference (1990) on Machine learning
Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time

Machine Learning
Exploration bonuses and dual control

Machine Learning
Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Control of exploitation-exploration meta-parameter in reinforcement learning

Neural Networks - Computational models of neuromodulation
Reinforcement Learning in Continuous Time and Space

Neural Computation
Planning and acting in partially observable stochastic domains

Artificial Intelligence
Model based Bayesian exploration

UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence

Reinforcement-learning agents with different temperature parameters explain the variety of human action-selection behavior in a Markov decision process task

Neurocomputing

Quantified Score

Hi-index	0.01

Visualization

Abstract

In this paper, we discuss an optimal decision-making problem in an unknown environment on the bases of both machine learning and brain learning. We present a model-based reinforcement learning (RL) in which the environment is directly estimated. Our RL performs action selection according to the detection of environmental changes and the current value function. In a partially-observable situation, in which the environment includes unobservable state variables, our RL incorporates estimation of unobservable variables. We propose a possible functional model of our RL, focusing on the prefrontal cortex and the anterior cingulate cortex. To test the model, we conducted a human imaging study during a sequential learning task, and found significant activations in the dorsolateral prefrontal cortex and the anterior cingulate cortex during RL. From a comparison of the mean activations in the earlier and later learning phases, we suggest that the dorsolateral prefrontal cortex maintains and manipulates the environmental model, while the anterior cingulate cortex is related to the uncertainty of action selection. These experimental results are consistent with our model.