An introduction to reinforcement learning theory: value function methods

• Authors: Peter L. Bartlett
• Venue: Advanced lectures on machine learning
• Year: 2003

These lecture notes are intended to give a tutorial introduction to the formulation and analysis of reinforcement learning problems. In these problems, an agent chooses actions to take in some environment, aiming to maximize a reward function. Many control, scheduling, planning and game-playing tasks can be formulated in this way, as problems of controlling a Markov decision process. We review the classical dynamic programming approaches to finding optimal controllers. For large state spaces, these techniques are impractical. We review methods based on approximate value functions, estimated via simulation. In particular, we discuss the motivation for (and shortcomings of) the TD (λ) algorithm.