Sample-efficient evolutionary function approximation for reinforcement learning

  • Authors:
  • Shimon Whiteson;Peter Stone

  • Affiliations:
  • Department of Computer Sciences, University of Texas at Austin, Austin, TX;Department of Computer Sciences, University of Texas at Austin, Austin, TX

  • Venue:
  • AAAI'06 Proceedings of the 21st national conference on Artificial intelligence - Volume 1
  • Year:
  • 2006

Quantified Score

Hi-index 0.00

Visualization

Abstract

Reinforcement learning problems are commonly tackled with temporal difference methods, which attempt to estimate the agent's optimal value function. In most real-world problems, learning this value function requires a function approximator, which maps state-action pairs to values via a concise, parameterized function. In practice, the success of function approximators depends on the ability of the human designer to select an appropriate representation for the value function. A recently developed approach called evolutionary function approximation uses evolutionary computation to automate the search for effective representations. While this approach can substantially improve the performance of TD methods, it requires many sample episodes to do so. We present an enhancement to evolutionary function approximation that makes it much more sample-efficient by exploiting the off-policy nature of certain TD methods. Empirical results in a server job scheduling domain demonstrate that the enhanced method can learn better policies than evolution or TD methods alone and can do so in many fewer episodes than standard evolutionary function approximation.