Multiresolution state-space discretization method for Q-learning

Authors:
Amanda Lampton;John Valasek
Affiliations:
Texas A&M University, College Station, Texas and Vehicle Systems & Control Laboratory, Aerospace Engineering Department;Texas A&M University, College Station, Texas and Vehicle Systems & Control Laboratory, Aerospace Engineering Department
Venue:
ACC'09 Proceedings of the 2009 conference on American Control Conference
Year:
2009

Citing 3
Cited 1

Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Improved Adaptive–Reinforcement Learning Control for Morphing Unmanned Air Vehicles

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics
Quad-Q-learning

IEEE Transactions on Neural Networks

Sparse gradient-based direct policy search

ICONIP'12 Proceedings of the 19th international conference on Neural Information Processing - Volume Part IV

Quantified Score

Hi-index	0.00

Visualization

Abstract

For large scale problems Q-Learning often suffers from the Curse of Dimensionality due to large numbers of possible state-action pairs. This paper develops a multiresolution state-space discretization method for the episodic unsupervised learning method of Q-Learning, in which a state-space is adaptively discretized by progressively finer grids around the areas of interest within the state or learning space. Optimality of the learning algorithm is addressed by a cost function. Applied to a morphing airfoil with two morphing parameters (two state variables), it is shown that by setting the multiresolution method to define the area of interest by the goal the agent seeks, this method can learn a specific goal within ±0.002, while reducing the total number of state-action pairs need to achieve this level of specificity by almost 90%.