Temporal difference learning and TD-Gammon
Communications of the ACM
Computational models of neuromodulation
Neural Computation
TD Models of reward predictive responses in dopamine neurons
Neural Networks - Computational models of neuromodulation
Learning to Predict by the Methods of Temporal Differences
Machine Learning
A Computational Model of How the Basal Ganglia Produce Sequences
Journal of Cognitive Neuroscience
Reinforcement learning: a survey
Journal of Artificial Intelligence Research
Metalearning and neuromodulation
Neural Networks - Computational models of neuromodulation
Actor-Critic Models of Reinforcement Learning in the Basal Ganglia: From Natural to Artificial Rats
Adaptive Behavior - Animals, Animats, Software Agents, Robots, Adaptive Systems
Representation and timing in theories of the dopamine system
Neural Computation
Modeling dopamine activity by Reinforcement Learning methods: implications from two recent models
Artificial Intelligence Review
Neural Correlates of Anticipation in Cerebellum, Basal Ganglia, and Hippocampus
Anticipatory Behavior in Adaptive Learning Systems
A Computational Model of Cortico-Striato-Thalamic Circuits in Goal-Directed Behaviour
ICANN '08 Proceedings of the 18th international conference on Artificial Neural Networks, Part II
Sustained activities and retrieval in a computational model of the perirhinal cortex
Journal of Cognitive Neuroscience
A spiking neural network model of an actor-critic learning agent
Neural Computation
Predictive models in the brain
Connection Science
Journal of Cognitive Neuroscience
Population models of temporal differentiation
Neural Computation
Why and how hippocampal transition cells can be used in reinforcement learning
SAB'10 Proceedings of the 11th international conference on Simulation of adaptive behavior: from animals to animats
Modeling basal ganglia for understanding parkinsonian reaching movements
Neural Computation
A model of reaching that integrates reinforcement learning and population encoding of postures
SAB'06 Proceedings of the 9th international conference on From Animals to Animats: simulation of Adaptive Behavior
Novel method for using Q-learning in small microcontrollers
Proceedings of the 51st ACM Southeast Conference
Strategic cognitive sequencing: a computational cognitive neuroscience approach
Computational Intelligence and Neuroscience - Special issue on Neurocognitive Models of Sense Making
Towards a real-time interface between a biomimetic model of sensorimotor cortex and a robotic arm
Pattern Recognition Letters
Hi-index | 0.01 |
A large number of computational models of information processing in the basal ganglia have been developed in recent years. Prominent in these are actor-critic models of basal ganglia functioning, which build on the strong resemblance between dopamine neuron activity and the temporal difference prediction error signal in the critic, and between dopamine-dependent long-term synaptic plasticity in the striatum and learning guided by a prediction error signal in the actor. We selectively review several actor-critic models of the basal ganglia with an emphasis on two important aspects: the way in which models of the critic reproduce the temporal dynamics of dopamine firing, and the extent to which models of the actor take into account known basal ganglia anatomy and physiology. To complement the efforts to relate basal ganglia mechanisms to reinforcement learning (RL), we introduce an alternative approach to modeling a critic network, which uses Evolutionary Computation techniques to 'evolve' an optimal RL mechanism, and relate the evolved mechanism to the basic model of the critic. We conclude our discussion of models of the critic by a critical discussion of the anatomical plausibility of implementations of a critic in basal ganglia circuitry, and conclude that such implementations build on assumptions that are inconsistent with the known anatomy of the basal ganglia. We return to the actor component of the actor-critic model, which is usually modeled at the striatal level with very little detail. We describe an alternative model of the basal ganglia which takes into account several important, and previously neglected, anatomical and physiological characteristics of basal ganglia-thalamocortical connectivity and suggests that the basal ganglia performs reinforcement-biased dimensionality reduction of cortical inputs. We further suggest that since such selective encoding may bias the representation at the level of the frontal cortex towards the selection of rewarded plans and actions, the reinforcement-driven dimensionality reduction framework may serve as a basis for basal ganglia actor models. We conclude with a short discussion of the dual role of the dopamine signal in RL and in behavioral switching.