Using mathematical programming to solve Factored Markov Decision Processes with Imprecise Probabilities

Authors:
Karina Valdivia Delgado;Leliane Nunes De Barros;Fabio Gagliardi Cozman;Scott Sanner
Affiliations:
Escola de Artes, Ciências e Humanidades, Universidade de São Paulo, SP, Brazil;Instituto de Matemática e Estatística, Universidade de São Paulo, SP, Brazil;Escola Politécnica, Universidade de São Paulo, SP, Brazil;NICTA and the Australian National University, Canberra ACT 2601, Australia
Venue:
International Journal of Approximate Reasoning
Year:
2011

Citing 22
Cited 0

The complexity of Markov decision processes

Mathematics of Operations Research
Convex two-level optimization

Mathematical Programming: Series A and B
A model for reasoning about persistence and causation

Computational Intelligence
An analysis of stochastic shortest path problems

Mathematics of Operations Research
Bounded-parameter Markov decision process

Artificial Intelligence
Computing Factored Value Functions for Policies in Structured MDPs

IJCAI '99 Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence
Planning under uncertainty in complex structured environments

Planning under uncertainty in complex structured environments
On Constraint Sampling in the Linear Programming Approach to Approximate Dynamic Programming

Mathematics of Operations Research
A Trust-Region Method for Nonlinear Bilevel Programming: Algorithm and Computational Experience

Computational Optimization and Applications
Linear approximations for factored markov decision processes

Linear approximations for factored markov decision processes
Symmetric approximate linear programming for factored MDPs with application to constrained problems

Annals of Mathematics and Artificial Intelligence
Robust Control of Markov Decision Processes with Uncertain Transition Matrices

Operations Research
Augmented Lagrangian methods under the constant positive linear dependence constraint qualification

Mathematical Programming: Series A and B
An inexact-restoration method for nonlinear bilevel programming problems

Computational Optimization and Applications
Samuel meets Amarel: automating value function approximation using global state space analysis

AAAI'05 Proceedings of the 20th national conference on Artificial intelligence - Volume 2
Planning under risk and Knightian uncertainty

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Discrete time Markov chains with interval probabilities

International Journal of Approximate Reasoning
Robust planning with (L)RTDP

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Graphical models for imprecise probabilities

International Journal of Approximate Reasoning
SPUDD: stochastic planning using decision diagrams

UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence
Solving uncertain markov decision problems: an interval-based method

ICNC'06 Proceedings of the Second international conference on Advances in Natural Computation - Volume Part II
The steepest descent direction for the nonlinear bilevel programming problem

Operations Research Letters

Quantified Score

Hi-index	0.01

Visualization

Abstract

This paper investigates Factored Markov Decision Processes with Imprecise Probabilities (MDPIPs); that is, Factored Markov Decision Processes (MDPs) where transition probabilities are imprecisely specified. We derive efficient approximate solutions for Factored MDPIPs based on mathematical programming. To do this, we extend previous linear programming approaches for linear approximations in Factored MDPs, resulting in a multilinear formulation for robust ''maximin'' linear approximations in Factored MDPIPs. By exploiting the factored structure in MDPIPs we are able to demonstrate orders of magnitude reduction in solution time over standard exact non-factored approaches, in exchange for relatively low approximation errors, on a difficult class of benchmark problems with millions of states.