A policy iteration algorithm for Markov decision processes skip-free in one direction
Proceedings of the 2nd international conference on Performance evaluation methodologies and tools
Quasi-Birth-Death Processes, Tree-Like QBDs, Probabilistic 1-Counter Automata, and Pushdown Systems
QEST '08 Proceedings of the 2008 Fifth International Conference on Quantitative Evaluation of Systems
One-counter Markov decision processes
SODA '10 Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms
STACS'06 Proceedings of the 23rd Annual conference on Theoretical Aspects of Computer Science
Recursive markov decision processes and recursive stochastic games
ICALP'05 Proceedings of the 32nd international conference on Automata, Languages and Programming
Reachability in recursive markov decision processes
CONCUR'06 Proceedings of the 17th international conference on Concurrency Theory
Minimizing expected termination time in one-counter markov decision processes
ICALP'12 Proceedings of the 39th international colloquium conference on Automata, Languages, and Programming - Volume Part II
Playing games with counter automata
RP'12 Proceedings of the 6th international conference on Reachability Problems
Determinacy and optimal strategies in infinite-state stochastic reachability games
Theoretical Computer Science
Analyzing probabilistic pushdown automata
Formal Methods in System Design
| Hi-index | 0.00 |
One-counter MDPs (OC-MDPs) and one-counter simple stochastic games (OC-SSGs) are 1-player, and 2-player turn-based zero-sum, stochastic games played on the transition graph of classic one-counter automata (equivalently, pushdown automata with a 1-letter stack alphabet). A key objective for the analysis and verification of these games is the termination objective, where the players aim to maximize (minimize, respectively) the probability of hitting counter value 0, starting at a given control state and given counter value. Recently [4,2], we studied qualitative decision problems ("is the optimal termination value = 1?") for OC-MDPs (and OC-SSGs) and showed them to be decidable in P-time (in NP∩coNP, respectively). However, quantitative decision and approximation problems ("is the optimal termination value ≥ p", or "approximate the termination value within ɛ") are far more challenging. This is so in part because optimal strategies may not exist, and because even when they do exist they can have a highly non-trivial structure. It thus remained open even whether any of these quantitative termination problems are computable. In this paper we show that all quantitative approximation problems for the termination value for OC-MDPs and OC-SSGs are computable. Specifically, given a OC-SSG, and given ɛ 0, we can compute a value ν that approximates the value of the OC-SSG termination game within additive error ɛ, and furthermore we can compute ɛ-optimal strategies for both players in the game. A key ingredient in our proofs is a subtle martingale, derived from solving certain LPs that we can associate with a maximizing OC-MDP. An application of Azuma's inequality on these martingales yields a computable bound for the "wealth" at which a "rich person's strategy" becomes ??-optimal for OC-MDPs.