Adaptive algorithms and stochastic approximations
Adaptive algorithms and stochastic approximations
Technical Note: \cal Q-Learning
Machine Learning
Asynchronous Stochastic Approximation and Q-Learning
Machine Learning
A method for discrete stochastic optimization
Management Science
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Introduction to Reinforcement Learning
Introduction to Reinforcement Learning
Neuro-Dynamic Programming
The Linear Programming Approach to Approximate Dynamic Programming
Operations Research
Handbook of Learning and Approximate Dynamic Programming (IEEE Press Series on Computational Intelligence)
Learning Algorithms for Separable Approximations of Discrete Stochastic Optimization Problems
Mathematics of Operations Research
Approximate Dynamic Programming: Solving the Curses of Dimensionality (Wiley Series in Probability and Statistics)
INFORMS Journal on Computing
Learning to act using real-time dynamic programming
Artificial Intelligence
Strong points of weak convergence: a study using RPA gradient estimation for automatic learning
Automatica (Journal of IFAC)
An algorithm for approximating piecewise linear concave functions from sample gradients
Operations Research Letters
Regression methods for pricing complex American-style options
IEEE Transactions on Neural Networks
| Hi-index | 0.00 |
In this article, we develop a stochastic approximation method to solve a monotone estimation problem and use this method to enhance the empirical performance of the Q-learning algorithm when applied to Markov decision problems with monotone value functions. We begin by considering a monotone estimation problem where we want to estimate the expectation of a random vector, η. We assume that the components of E {η} are known to be in increasing order. The stochastic approximation method that we propose is designed to exploit this information by projecting its iterates onto the set of vectors with increasing components. The novel aspect of the method is that it uses projections with respect to the max norm. We show the almost sure convergence of the stochastic approximation method. After this result, we consider the Q-learning algorithm when applied to Markov decision problems with monotone value functions. We study a variant of the Q-learning algorithm that uses projections to ensure that the value function approximation obtained at each iteration is also monotone. Computational results indicate that the performance of the Q-learning algorithm can be improved significantly by exploiting the monotonicity property of the value functions.