Solving factored MDPs with hybrid state and action variables

Authors:
Branislav Kveton;Milos Hauskrecht;Carlos Guestrin
Affiliations:
Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA;Department of Computer Science, University of Pittsburgh, Pittsburgh, PA;Machine Learning Department and Computer Science Department, Carnegie Mellon University, Pittsburgh, PA
Venue:
Journal of Artificial Intelligence Research
Year:
2006

Citing 37
Cited 8

A model for reasoning about persistence and causation

Computational Intelligence
Practical Issues in Temporal Difference Learning

Machine Learning
TD-Gammon, a self-teaching backgammon program, achieves master-level play

Neural Computation
Temporal difference learning and TD-Gammon

Communications of the ACM
A counterexample to temporal differences learning

Neural Computation
Markov Decision Processes: Discrete Stochastic Dynamic Programming

Markov Decision Processes: Discrete Stochastic Dynamic Programming
Introduction to Linear Optimization

Introduction to Linear Optimization
Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Neuro-Dynamic Programming

Neuro-Dynamic Programming
Variable Resolution Discretization in Optimal Control

Machine Learning
Computing Factored Value Functions for Policies in Structured MDPs

IJCAI '99 Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence
Approximating MAP using Local Search

UAI '01 Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence
Context-specific multiagent coordination and planning with factored MDPs

Eighteenth national conference on Artificial intelligence
Greedy linear value-approximation for factored Markov decision processes

Eighteenth national conference on Artificial intelligence
Dynamic Programming

Dynamic Programming
Approximate solutions to markov decision processes

Approximate solutions to markov decision processes
Selecting approximately-optimal actions in complex structured domains

Selecting approximately-optimal actions in complex structured domains
The Linear Programming Approach to Approximate Dynamic Programming

Operations Research
Planning under uncertainty in complex structured environments

Planning under uncertainty in complex structured environments
On Constraint Sampling in the Linear Programming Approach to Approximate Dynamic Programming

Mathematics of Operations Research
Dynamic programming for structured continuous Markov decision problems

UAI '04 Proceedings of the 20th conference on Uncertainty in artificial intelligence
Metrics for finite Markov decision processes

UAI '04 Proceedings of the 20th conference on Uncertainty in artificial intelligence
Solving factored MDPs with continuous and discrete variables

UAI '04 Proceedings of the 20th conference on Uncertainty in artificial intelligence
Annealed MAP

UAI '04 Proceedings of the 20th conference on Uncertainty in artificial intelligence
Learning basis functions in hybrid domains

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Learning representation and control in continuous Markov decision processes

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Samuel meets Amarel: automating value function approximation using global state space analysis

AAAI'05 Proceedings of the 20th national conference on Artificial intelligence - Volume 2
Value-function approximations for partially observable Markov decision processes

Journal of Artificial Intelligence Research
Efficient solution algorithms for factored MDPs

Journal of Artificial Intelligence Research
Generalizing plans to new environments in relational MDPs

IJCAI'03 Proceedings of the 18th international joint conference on Artificial intelligence
Max-norm projections for factored MDPs

IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 1
An MCMC approach to solving hybrid factored MDPs

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Exploiting structure in policy construction

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
A reinforcement learning approach to job-shop scheduling

IJCAI'95 Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2
Planning under continuous time and resource uncertainty: a challenge for AI

UAI'02 Proceedings of the Eighteenth conference on Uncertainty in artificial intelligence
Bucket elimination: a unifying framework for probabilistic inference

UAI'96 Proceedings of the Twelfth international conference on Uncertainty in artificial intelligence
Solving MAP exactly using systematic search

UAI'03 Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence

Adaptive multi-robot wide-area exploration and mapping

Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 1
Factored value iteration converges

Acta Cybernetica
Factored temporal difference learning in the new ties environment

Acta Cybernetica
A heuristic search approach to planning with continuous resources in stochastic domains

Journal of Artificial Intelligence Research
Computational approaches to reachability analysis of stochastic hybrid systems

HSCC'07 Proceedings of the 10th international conference on Hybrid systems: computation and control
Planning in stochastic domains for multiple agents with individual continuous resource state-spaces

Autonomous Agents and Multi-Agent Systems
A framework and a mean-field algorithm for the local control of spatial processes

International Journal of Approximate Reasoning
Robust optimization for hybrid MDPs with state-dependent noise

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence

Quantified Score

Hi-index	0.00

Visualization

Abstract

Efficient representations and solutions for large decision problems with continuous and discrete variables are among the most important challenges faced by the designers of automated decision support systems. In this paper, we describe a novel hybrid factored Markov decision process (MDP) model that allows for a compact representation of these problems, and a new hybrid approximate linear programming (HALP) framework that permits their efficient solutions. The central idea of HALP is to approximate the optimal value function by a linear combination of basis functions and optimize its weights by linear programming. We analyze both theoretical and computational aspects of this approach, and demonstrate its scale-up potential on several hybrid optimization problems.