Polynomial time algorithms for branching markov decision processes and probabilistic min(max) polynomial bellman equations

  • Authors:

  • Kousha Etessami;Alistair Stewart;Mihalis Yannakakis

  • Affiliations:

  • School of Informatics, University of Edinburgh, UK;School of Informatics, University of Edinburgh, UK;Department of Computer Science, Columbia University

  • Venue:
  • ICALP'12 Proceedings of the 39th international colloquium conference on Automata, Languages, and Programming - Volume Part I
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

We show that one can approximate the least fixed point solution for a multivariate system of monotone probabilistic max (min) polynomial equations, in time polynomial in both the encoding size of the system of equations and in log(1/ε), where ε0 is the desired additive error bound of the solution. (The model of computation is the standard Turing machine model.) These equations form the Bellman optimality equations for several important classes of infinite-state Markov Decision Processes (MDPs). Thus, as a corollary, we obtain the first polynomial time algorithms for computing to within arbitrary desired precision the optimal value vector for several classes of infinite-state MDPs which arise as extensions of classic, and heavily studied, purely stochastic processes. These include both the problem of maximizing and minimizing the termination (extinction) probability of multi-type branching MDPs, stochastic context-free MDPs, and 1-exit Recursive MDPs. We also show that we can compute in P-time an ε-optimal policy for any given desired ε0.