A Unified Analysis of Value-Function-Based Reinforcement Learning Algorithms

  • Authors:
  • Csaba Szepesvári;Michael L. Littman

  • Affiliations:
  • Mindmaker, Ltd., Budapest 1121, Konkoly Thege M. U. 29-33, Hungary;Department of Computer Science, Duke University, Durham, NC 27708-0129, U.S.A.

  • Venue:
  • Neural Computation
  • Year:
  • 1999

Quantified Score

Hi-index 0.00

Visualization

Abstract

Reinforcement learning is the problem of generating optimal behavior in a sequential decision-making environment given the opportunity of interacting with it. Many algorithms for solving reinforcement-learning problems work by computing improved estimates of the optimal value function. We extend prior analyses of reinforcement-learning algorithms and present a powerful new theorem that can provide a unified analysis of such value-function-based reinforcement-learning algorithms. The usefulness of the theorem lies in how it allows the convergence of a complex asynchronous reinforcement-learning algorithm to be proved by verifying that a simpler synchronous algorithm converges. We illustrate the application of the theorem by analyzing the convergence of Q-learning, model-based reinforcement learning, Q-learning with multistate updates, Q-learning for Markov games, and risk-sensitive reinforcement learning.