Approximate dynamic programming using Bellman residual elimination and Gaussian process regression

Authors:
Brett Bethke;Jonathan P. How
Affiliations:
Dept. of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA;Aeronautics and Astronautics, MIT
Venue:
ACC'09 Proceedings of the 2009 conference on American Control Conference
Year:
2009

Citing 9
Cited 1

Temporal difference learning and TD-Gammon

Communications of the ACM
Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Neuro-Dynamic Programming

Neuro-Dynamic Programming
A Tutorial on Support Vector Machines for Pattern Recognition

Data Mining and Knowledge Discovery
Least-squares policy iteration

The Journal of Machine Learning Research
Fast direct policy evaluation using multiscale analysis of Markov diffusion processes

ICML '06 Proceedings of the 23rd international conference on Machine learning
Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)

Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)
Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path

Machine Learning
Dynamic Programming and Optimal Control, Vol. II

Dynamic Programming and Optimal Control, Vol. II

PH optimal control in the clarifying process of sugar cane juice based on DHP

ICIC'10 Proceedings of the 6th international conference on Advanced intelligent computing theories and applications: intelligent computing

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents an approximate policy iteration algorithm for solving infinite-horizon, discounted Markov decision processes (MDPs) for which a model of the system is available. The algorithm is similar in spirit to Bellman residual minimization methods. However, by using Gaussian process regression with nondegenerate kernel functions as the underlying cost-to-go function approximation architecture, the algorithm is able to explicitly construct cost-to-go solutions for which the Bellman residuals are identically zero at a set of chosen sample states. For this reason, we have named our approach Bellman residual elimination (BRE). Since the Bellman residuals are zero at the sample states, our BRE algorithm can be proven to reduce to exact policy iteration in the limit of sampling the entire state space. Furthermore, the algorithm can automatically optimize the choice of any free kernel parameters and provide error bounds on the resulting cost-to-go solution. Computational results on a classic reinforcement learning problem indicate that the algorithm yields a high-quality policy and cost approximation.