An Approximation Approach for the Deviation Matrix of Continuous-Time Markov Processes with Application to Markov Decision Theory

Authors:
Nicole Leder;Bernd Heidergott;Arie Hordijk
Affiliations:
Department of Mathematics, University of Hamburg, Hamburg 20146, Germany;Department of Econometrics and Operations Research, and Tinbergen Institute, Vrije Universiteit Amsterdam, Amsterdam 1081 HV, The Netherlands;Mathematical Institute, Leiden University, Leiden 2300 RA, The Netherlands
Venue:
Operations Research
Year:
2010

Citing 15
Cited 2

Average, sensitive and Blackwell optimal policies in denumerable Markov decisionchains with unbounded rewards

Mathematics of Operations Research
Denumerable semi-Markov decision chains with small interest rates

Annals of Operations Research
Recurrence conditions for average and Blackwell optimality in denumerable state Markov decision chains

Mathematics of Operations Research
Discrete-time controlled Markov processes with average cost criterion: a survey

SIAM Journal on Control and Optimization
On the relation between recurrence and ergodicity properties in denumerable Markov decision chains

Mathematics of Operations Research
Contraction conditions for average and &agr;-discount optimality in countable state Markov games with unbounded rewards

Mathematics of Operations Research
M/G/1-Type Markov Processes: A Tutorial

Performance Evaluation of Complex Systems: Techniques and Tools, Performance 2002, Tutorial Lectures
THE DEVIATION MATRIX OF A CONTINUOUS-TIME MARKOV CHAIN

Probability in the Engineering and Informational Sciences
Optimal Control of Ergodic Continuous-Time Markov Chains with Average Sample-Path Rewards

SIAM Journal on Control and Optimization
Stochastic Learning and Optimization: A Sensitivity-Based Approach (International Series on Discrete Event Dynamic Systems)

Stochastic Learning and Optimization: A Sensitivity-Based Approach (International Series on Discrete Event Dynamic Systems)
Discrete-Event Control of Stochastic Networks: Multimodularity and Regularity (Lecture Notes in Mathematics)

Discrete-Event Control of Stochastic Networks: Multimodularity and Regularity (Lecture Notes in Mathematics)
Continuous-Time Markov Decision Processes with Discounted Rewards: The Case of Polish Spaces

Mathematics of Operations Research
Series Expansions For Finite-State Markov Chains

Probability in the Engineering and Informational Sciences
Brief paper: Policy iteration based feedback control

Automatica (Journal of IFAC)
Series Expansions for Continuous-Time Markov Processes

Operations Research

Error bounds on practical approximation for two tandem queue with blocking and non-preemptive priority

Computers & Mathematics with Applications
A functional approximation for the M/G/1/N queue

Discrete Event Dynamic Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present an update formula that allows the expression of the deviation matrix of a continuous-time Markov process with denumerable state space having generator matrix Q* through a continuous-time Markov process with generator matrix Q. We show that under suitable stability conditions the algorithm converges at a geometric rate. By applying the concept to three different examples, namely, the M/M/1 queue with vacations, the M/G/1 queue, and a tandem network, we illustrate the broad applicability of our approach. For a problem in admission control, we apply our approximation algorithm to Markov decision theory for computing the optimal control policy. Numerical examples are presented to highlight the efficiency of the proposed algorithm.