Policy Gradient in Continuous Time
The Journal of Machine Learning Research
Journal of Computational and Applied Mathematics
| Hi-index | 0.00 |
We consider a multidimensional diffusion process $(X^\alpha_t)_{0\leq t\leq T}$ whose dynamics depends on a parameter $\alpha$. Our first purpose is to write as an expectation the sensitivity $\nabla_\alpha J(\alpha)$ for the expected cost $J(\alpha)=\mathbb{E}(f(X^\alpha_T))$, in order to evaluate it using Monte Carlo simulations. This issue arises, for example, from stochastic control problems (where the controller is parameterized, which reduces the control problem to a parametric optimization one) or from model misspecifications in finance. Previous evaluations of $\nabla_\alpha J(\alpha)$ using simulations were limited to smooth cost functions $f$ or to diffusion coefficients not depending on $\alpha$ (see Yang and Kushner, SIAM J. Control Optim., 29 (1991), pp. 1216--1249). In this paper, we cover the general case, deriving three new approaches to evaluate $\nabla_\alpha J(\alpha)$, which we call the Malliavin calculus approach, the adjoint approach, and the martingale approach. To accomplish this, we leverage Itô calculus, Malliavin calculus, and martingale arguments. In the second part of this work, we provide discretization procedures to simulate the relevant random variables; then we analyze their respective errors. This analysis proves that the discretization error is essentially linear with respect to the time step. This result, which was already known in some specific situations, appears to be true in this much wider context. Finally, we provide numerical experiments in random mechanics and finance and compare the different methods in terms of variance, complexity, computational time, and time discretization error.