Automata, Probability, and Recursion
CIAA '08 Proceedings of the 13th international conference on Implementation and Applications of Automata
Computing relaxed abstract semantics w.r.t. quadratic zones precisely
SAS'10 Proceedings of the 17th international conference on Static analysis
Solving systems of rational equations through strategy iteration
ACM Transactions on Programming Languages and Systems (TOPLAS)
Qualitative reachability in stochastic BPA games
Information and Computation
ICALP'12 Proceedings of the 39th international colloquium conference on Automata, Languages, and Programming - Volume Part I
Minimizing expected termination time in one-counter markov decision processes
ICALP'12 Proceedings of the 39th international colloquium conference on Automata, Languages, and Programming - Volume Part II
Playing games with counter automata
RP'12 Proceedings of the 6th international conference on Reachability Problems
Stochastic parity games on lossy channel systems
QEST'13 Proceedings of the 10th international conference on Quantitative Evaluation of Systems
Hi-index | 0.00 |
We study the complexity of a class of Markov decision processes and, more generally, stochastic games, called 1-exit Recursive Markov Decision Processes (1-RMDPs) and Simple Stochastic Games (1-RSSGs) with strictly positive rewards. These are a class of finitely presented countable-state zero-sum stochastic games, with total expected reward objective. They subsume standard finite-state MDPs and Condon's simple stochastic games and correspond to optimization and game versions of several classic stochastic models, with rewards. Such stochastic models arise naturally as models of probabilistic procedural programs with recursion, and the problems we address are motivated by the goal of analyzing the optimal/pessimal expected running time in such a setting.We give polynomial time algorithms for 1-exit Recursive Markov decision processes (1-RMDPs) with positive rewards. Specifically, we show that the exact optimal value of both maximizing and minimizing 1-RMDPs with positive rewards can be computed in polynomial time (this value may be 驴). For two-player 1-RSSGs with positive rewards, we prove a "stackless and memoryless" determinacy result, and show that deciding whether the game value is at least a given value r is in NP 驴 coNP. We also prove that a simultaneous strategy improvement algorithm converges to the value and optimal strategies for these stochastic games. We observe that 1-RSSG positive reward games are "harder" than finite-state SSGs in several senses.