Approximate Verification of the Symbolic Dynamics of Markov Chains
LICS '12 Proceedings of the 2012 27th Annual IEEE/ACM Symposium on Logic in Computer Science
The steady-state control problem for markov decision processes
QEST'13 Proceedings of the 10th international conference on Quantitative Evaluation of Systems
| Hi-index | 0.01 |
The semantics of Markov Decision Processes (MDPs), when viewed as transformers of probability distributions, can described as a labeled transition system over the probability distributions over the states of the MDP. The MDP can be seen as defining a set of executions, where each execution is a sequence of probability distributions. Reasoning about sequences of distributions allows one to express properties not expressible in logics like PCTL, examples include expressing bounds on transient rewards and expected values of random variables, as well as comparing the probability of being in one set of states at a given time with another set of states. With respect to such a semantics, the problem of checking that the MDP never reaches a bad distribution is undecidable~\cite{qest10}. In this paper, we identify a special class of MDPs called \emph{semi-regular} MDPs that have a unique non-empty, compact, invariant set of distributions, for which we show that checking any $\omega$-regular property is decidable. Our decidability result also implies that for semi-regular probabilistic finite automata with isolated cut-points, the emptiness problem is decidable.