Competitive Markov decision processes
Competitive Markov decision processes
Modeling and analysis of stochastic systems
Modeling and analysis of stochastic systems
Stochastic dynamic programming and the control of queueing systems
Stochastic dynamic programming and the control of queueing systems
Sequencing Tasks with Exponential Service Times to Minimize the Expected Flow Time or Makespan
Journal of the ACM (JACM)
Model-checking continuous-time Markov chains
ACM Transactions on Computational Logic (TOCL)
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Continuous Time Discounted Jump Markov Decision Processes: A Discrete-Event Approach
Mathematics of Operations Research
Theoretical Computer Science - Tools and algorithms for the construction and analysis of systems (TACAS 2004)
Delayed Nondeterminism in Continuous-Time Markov Decision Processes
FOSSACS '09 Proceedings of the 12th International Conference on Foundations of Software Science and Computational Structures: Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2009
Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes
QEST '10 Proceedings of the 2010 Seventh International Conference on the Quantitative Evaluation of Systems
Model checking algorithms for CTMDPs
CAV'11 Proceedings of the 23rd international conference on Computer aided verification
A characterization of meaningful schedulers for continuous-time markov decision processes
FORMATS'06 Proceedings of the 4th international conference on Formal Modeling and Analysis of Timed Systems
| Hi-index | 5.23 |
We study time-bounded reachability in continuous-time Markov decision processes (CTMDPs) and games (CTGs) for time-abstract scheduler classes. Reachability problems play a paramount role in probabilistic model checking. Consequently, their analysis has been studied intensively, and approximation techniques are well understood. From a mathematical point of view, however, the question of approximation is secondary compared to the fundamental question whether or not optimal control exists. In this article, we demonstrate the existence of optimal schedulers for the time-abstract scheduler classes for CTMDPs. For CTGs, we distinguish two cases: the simple case where both players face the same restriction to use time-abstract strategies (symmetric CTGs) and the case where one player is a completely informed adversary (asymmetric CTGs). While for the former case optimal strategies exist, we prove that for asymmetric CTGs there is not necessarily a scheduler that attains the optimum. It turns out that for CTMDPs and symmetric CTGs optimal time-abstract schedulers have an amazingly simple structure: they converge to a memoryless scheduling policy after a finite number of steps. This allows us to compute time-abstract strategies with finite memory.