Online actor critic algorithm to solve the continuous-time infinite horizon optimal control problem

Authors:
Kyriakos G. Vamvoudakis;Frank L. Lewis
Affiliations:
Automation and Robotics Research Institute, University of Texas at Arlington, Fort Worth, TX;Automation and Robotics Research Institute, University of Texas at Arlington, Fort Worth, TX
Venue:
IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Year:
2009

Citing 12
Cited 2

Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation

Automatica (Journal of IFAC)
Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Neural Network Control of Robot Manipulators and Nonlinear Systems

Neural Network Control of Robot Manipulators and Nonlinear Systems
Neuro-Dynamic Programming

Neuro-Dynamic Programming
Handbook of Learning and Approximate Dynamic Programming (IEEE Press Series on Computational Intelligence)

Handbook of Learning and Approximate Dynamic Programming (IEEE Press Series on Computational Intelligence)
Reinforcement Learning in Continuous Time and Space

Neural Computation
Brief paper: Adaptive optimal control for continuous-time linear systems based on policy iteration

Automatica (Journal of IFAC)
Adaptive dynamic programming

IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews
Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach

Automatica (Journal of IFAC)
Adaptive critic designs

IEEE Transactions on Neural Networks
Continuous-Time Adaptive Critics

IEEE Transactions on Neural Networks
Neural net robot controller with guaranteed tracking performance

IEEE Transactions on Neural Networks

A three-network architecture for on-line learning and optimization based on adaptive dynamic programming

Neurocomputing
2012 Special Issue: A boundedness result for the direct heuristic dynamic programming

Neural Networks

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we discuss an online algorithm based on policy iteration for learning the continuous-time (CT) optimal control solution with infinite horizon cost for nonlinear systems with known dynamics. We present an online adaptive algorithm implemented as an actor/critic structure which involves simultaneous continuous-time adaptation of both actor and critic neural networks. We call this 'synchronous' policy iteration. A persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function. Novel tuning algorithms are given for both critic and actor networks, with extra terms in the actor tuning law being required to guarantee closed-loop dynamical stability. The convergence to the optimal controller is proven, and stability of the system is also guaranteed. Simulation exam pies show the effectiveness of the new algorithm.