The Complexity of Decentralized Control of Markov Decision Processes

Authors:
Daniel S. Bernstein;Robert Givan;Neil Immerman;Shlomo Zilberstein
Affiliations:
-;-;-;-
Venue:
Mathematics of Operations Research
Year:
2002

Citing 13
Cited 133

Intractable problems in control theory

SIAM Journal on Control and Optimization
The complexity of Markov decision processes

Mathematics of Operations Research
Planning and acting in partially observable stochastic domains

Artificial Intelligence
On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems

AAAI '99/IAAI '99 Proceedings of the sixteenth national conference on Artificial intelligence and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence
Complexity of finite-horizon Markov decision process problems

Journal of the ACM (JACM)
Distributed Value Functions

ICML '99 Proceedings of the Sixteenth International Conference on Machine Learning
Learning to Cooperate via Policy Search

UAI '00 Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence
Algorithms for partially observable markov decision processes

Algorithms for partially observable markov decision processes
My brain is full: when more memory helps

UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence
Solving POMDPs by searching the space of finite policies

UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence
Solving POMDPs by searching in policy space

UAI'98 Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence
Incremental pruning: a simple, fast, exact method for partially observable Markov decision processes

UAI'97 Proceedings of the Thirteenth conference on Uncertainty in artificial intelligence
Survey A survey of computational complexity results in systems and control

Automatica (Journal of IFAC)

Transition-independent decentralized markov decision processes

AAMAS '03 Proceedings of the second international joint conference on Autonomous agents and multiagent systems
Optimizing information exchange in cooperative multi-agent systems

AAMAS '03 Proceedings of the second international joint conference on Autonomous agents and multiagent systems
Minimizing communication cost in a distributed Bayesian network using a decentralized MDP

AAMAS '03 Proceedings of the second international joint conference on Autonomous agents and multiagent systems
The complexity of multiagent systems: the price of silence

AAMAS '03 Proceedings of the second international joint conference on Autonomous agents and multiagent systems
Decentralized Markov Decision Processes with Event-Driven Interactions

AAMAS '04 Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems - Volume 1
Decentralized Language Learning through Acting

AAMAS '04 Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems - Volume 3
Interactive POMDPs: Properties and Preliminary Results

AAMAS '04 Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems - Volume 3
Multiagent coordination by Extended Markov Tracking

Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems
Communication management using abstraction in distributed bayesian networks

Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems
Playing games in many possible worlds

EC '06 Proceedings of the 7th ACM conference on Electronic commerce
Decentralized planning under uncertainty for teams of communicating agents

AAMAS '06 Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems
Stochastic planning for weakly-coupled distributed agents

AAMAS '06 Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems
Agent interaction in distributed POMDPs and its implications on complexity

AAMAS '06 Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems
Communication management using abstraction in distributed Bayesian networks

AAMAS '06 Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems
Exact solutions of interactive POMDPs using behavioral equivalence

AAMAS '06 Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems
Learning to communicate in a decentralized environment

Autonomous Agents and Multi-Agent Systems
Local strategy learning in networked multi-agent team formation

Autonomous Agents and Multi-Agent Systems
Shaping multi-agent systems with gradient reinforcement learning

Autonomous Agents and Multi-Agent Systems
Exploiting factored representations for decentralized execution in multiagent teams

Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems
Q-value functions for decentralized POMDPs

Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems
Distributed intrusion detection in partially observable Markov decision processes

Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems
Subjective approximate solutions for decentralized POMDPs

Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems
Modeling plan coordination in multiagent decision processes

Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems
Value-based observation compression for DEC-POMDPs

Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 1
Exploiting locality of interaction in factored Dec-POMDPs

Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 1
Interaction-driven Markov games for decentralized multiagent planning under uncertainty

Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 1
Reinforcement learning for DEC-MDPs with changing action sets and partially ordered dependencies

Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 3
Solving Decentralized Continuous Markov Decision Problems with Structured Reward

KI '07 Proceedings of the 30th annual German conference on Advances in Artificial Intelligence
Solving Large-Scale and Sparse-Reward DEC-POMDPs with Correlation-MDPs

RoboCup 2007: Robot Soccer World Cup XI
Joint Equilibrium Policy Search for Multi-Agent Scheduling Problems

MATES '08 Proceedings of the 6th German conference on Multiagent System Technologies
Towards the Self-regulation of Personality-Based Social Exchange Processes in Multiagent Systems

SBIA '08 Proceedings of the 19th Brazilian Symposium on Artificial Intelligence: Advances in Artificial Intelligence
Evaluation of Batch-Mode Reinforcement Learning Methods for Solving DEC-MDPs with Changing Action Sets

Recent Advances in Reinforcement Learning
Commitment-based service coordination

International Journal of Agent-Oriented Software Engineering
Constraint-based dynamic programming for decentralized POMDPs with structured interactions

Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Point-based incremental pruning heuristic for solving finite-horizon DEC-POMDPs

Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Lossless clustering of histories in decentralized POMDPs

Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Learning of coordination: exploiting sparse interactions in multiagent systems

Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2
Exploiting locality of interactions using a policy-gradient approach in multiagent learning

Proceedings of the 2008 conference on ECAI 2008: 18th European Conference on Artificial Intelligence
Military network security using self organized multi-agent entangled hierarchies

Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers
Self organized multi-agent entangled hierarchies for network security

Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers
Contribution to the Control of a MAS's Global Behaviour: Reinforcement Learning Tools

Engineering Societies in the Agents World IX
Dynamic programming for partially observable stochastic games

AAAI'04 Proceedings of the 19th national conference on Artifical intelligence
A framework for optimal sequential planning in multiagent settings

AAAI'04 Proceedings of the 19th national conference on Artifical intelligence
An iterative algorithm for solving constrained decentralized Markov decision processes

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Point-based dynamic programming for DEC-POMDPs

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Population and agent based models for language convergence

AAAI'06 proceedings of the 21st national conference on Artificial intelligence - Volume 2
Agent influence as a predictor of difficulty for decentralized problem-solving

AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 1
Partially-synchronized DEC-MDPs in dynamic mechanism design

AAAI'08 Proceedings of the 23rd national conference on Artificial intelligence - Volume 1
Optimal multi-agent scheduling with constraint programming

IAAI'07 Proceedings of the 19th national conference on Innovative applications of artificial intelligence - Volume 2
Exploiting symmetries in POMDPs for point-based algorithms

AAAI'08 Proceedings of the 23rd national conference on Artificial intelligence - Volume 2
Interaction structure and dimensionality reduction in decentralized MDPs

AAAI'08 Proceedings of the 23rd national conference on Artificial intelligence - Volume 3
Decentralized control of cooperative systems: categorization and complexity analysis

Journal of Artificial Intelligence Research
Solving transition independent decentralized Markov decision processes

Journal of Artificial Intelligence Research
A framework for sequential planning in multi-agent settings

Journal of Artificial Intelligence Research
Cooperative information sharing to improve distributed learning in multi-agent systems

Journal of Artificial Intelligence Research
Communication-based decomposition mechanisms for decentralized MDPs

Journal of Artificial Intelligence Research
Optimal and approximate Q-value functions for decentralized POMDPs

Journal of Artificial Intelligence Research
Policy iteration for decentralized control of Markov decision processes

Journal of Artificial Intelligence Research
Monte Carlo sampling methods for approximating interactive POMDPs

Journal of Artificial Intelligence Research
Globally Optimal Multi-agent Reinforcement Learning Parameters in Distributed Task Assignment

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 02
Offline Planning for Communication by Exploiting Structured Interactions in Decentralized MDPs

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 02
Agent Influence and Intelligent Approximation in Multiagent Problems

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 02
Myopic and Non-myopic Communication under Partial Observability

WI-IAT '09 Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 02
Bounded policy iteration for decentralized POMDPs

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Language learning in multi-agent systems

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Planning for weakly-coupled partially observable stochastic games

IJCAI'05 Proceedings of the 19th international joint conference on Artificial intelligence
Event-detecting multi-agent MDPs: complexity and constant-factor approximation

IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
Delay-sensitive distributed power and transmission threshold control for S-ALOHA network with finite state Markov fading channels

IEEE Transactions on Wireless Communications
Review article: Synergizing reinforcement learning and game theory-A new direction for control

Applied Soft Computing
Transfer Learning for Reinforcement Learning Domains: A Survey

The Journal of Machine Learning Research
Self-organization for coordinating decentralized reinforcement learning

Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 1
Point-based backup for decentralized POMDPs: complexity and new algorithms

Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 1
The multi variable multi constrained distributed constraint optimization framework

Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 1
From policies to influences: a framework for nonlocal abstraction in transition-dependent Dec-POMDP agents

Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 1
Coordinated learning in multiagent MDPs with infinite state-space

Autonomous Agents and Multi-Agent Systems
An investigation into mathematical programming for finite horizon decentralized POMDPs

Journal of Artificial Intelligence Research
Point-based bounded policy iteration for decentralized POMDPs

PRICAI'10 Proceedings of the 11th Pacific Rim international conference on Trends in artificial intelligence
Planning in stochastic domains for multiple agents with individual continuous resource state-spaces

Autonomous Agents and Multi-Agent Systems
Decentralized MDPs with sparse interactions

Artificial Intelligence
CoUAV: a multi-UAV cooperative search path planning simulation environment

Proceedings of the 2010 Summer Computer Simulation Conference
Decentralized monitoring of distributed anytime algorithms

The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Cognitive policy learner: biasing winning or losing strategies

The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 2
Toward error-bounded algorithms for infinite-horizon DEC-POMDPs

The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 3
Distributed model shaping for scaling to decentralized POMDPs with hundreds of agents

The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 3
Message-passing algorithms for large structured decentralized POMDPs

The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 3
Solving efficiently Decentralized MDPs with temporal and resource constraints

Autonomous Agents and Multi-Agent Systems
On the power of global reward signals in reinforcement learning

MATES'11 Proceedings of the 9th German conference on Multiagent system technologies
Social Model Shaping for Solving Generic DEC-POMDPs

WI-IAT '11 Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Volume 02
Using Markov Decision Processes to define an adaptive strategy to control the spread of an animal disease

Computers and Electronics in Agriculture
Coordination of cooperation policies in a peer-to-peer system using swarm-based RL

Journal of Network and Computer Applications
An optimal best-first search algorithm for solving infinite horizon DEC-POMDPs

ECML'05 Proceedings of the 16th European conference on Machine Learning
The complexity of finding an optimal policy for language convergence

SAB'06 Proceedings of the 9th international conference on From Animals to Animats: simulation of Adaptive Behavior
Exploiting symmetries for single- and multi-agent Partially Observable Stochastic Domains

Artificial Intelligence
A POMDP model for guiding taxi cruising in a congested urban city

MICAI'11 Proceedings of the 10th Mexican international conference on Advances in Artificial Intelligence - Volume Part I
A convergent multiagent reinforcement learning approach for a subclass of cooperative stochastic games

ALA'11 Proceedings of the 11th international conference on Adaptive and Learning Agents
Topological value iteration algorithms

Journal of Artificial Intelligence Research
Continuous time planning for multiagent teams with temporal constraints

IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence - Volume Volume One
Scalable multiagent planning using probabilistic inference

IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence - Volume Volume Three
Active visual sensing and collaboration on mobile robots using hierarchical POMDPs

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
A decision-theoretic characterization of organizational influences

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1
Heuristic search of multiagent influence space

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 2
Generalized and bounded policy iteration for finitely-nested interactive POMDPs: scaling up

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 2
Planning and evaluating multiagent influences under reward uncertainty

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 3
Agent-human coordination with communication costs under uncertainty

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 3
GaTAC: a scalable and realistic testbed for multiagent decision making (demonstration)

Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 3
Multiagent decision by partial evaluation

Canadian AI'12 Proceedings of the 25th Canadian conference on Advances in Artificial Intelligence
Exploiting model equivalences for solving interactive dynamic influence diagrams

Journal of Artificial Intelligence Research
Exploiting independent relationships in multiagent systems for coordinated learning

PRICAI'12 Proceedings of the 12th Pacific Rim international conference on Trends in Artificial Intelligence
Q-Tree: automatic construction of hierarchical state representation for reinforcement learning

ICIRA'12 Proceedings of the 5th international conference on Intelligent Robotics and Applications - Volume Part III
Observer effect from stateful resources in agent sensing

Autonomous Agents and Multi-Agent Systems
Exploring the Importance of Information Relevance, Ontology and Utilities for Scalable Multi-agent Coordination

WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 02
Organizational design principles and techniques for decision-theoretic agents

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Producing efficient error-bounded solutions for transition independent decentralized mdps

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Approximate solutions for factored Dec-POMDPs with many agents

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Using conflict resolution to inform decentralized learning

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Distributed relational temporal difference learning

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Coordinating multi-agent reinforcement learning with limited communication

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Multiagent POMDPs with asynchronous execution

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Concurrent reinforcement learning as a rehearsal for decentralized planning under uncertainty

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
NetArg: an agent-based social simulator with argumentative agents

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Information sharing for care coordination

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
Reinforcement learning for decentralized planning under uncertainty

Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems
TCP ex machina: computer-generated congestion control

Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM
Decentralized multi-robot cooperation with auctioned POMDPs

International Journal of Robotics Research
Incremental clustering and expansion for faster optimal planning in decentralized POMDPs

Journal of Artificial Intelligence Research
ACTIDS: an active strategy for detecting and localizing network attacks

Proceedings of the 2013 ACM workshop on Artificial intelligence and security
Optimally solving dec-POMDPs as continuous-state MDPs

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Sufficient plan-time statistics for decentralized POMDPs

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Monte-Carlo expectation maximization for decentralized POMDPs

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Automated generation of interaction graphs for value-factored dec-POMDPs

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Decentralized Guidance Control of UAVs with Explicit Optimization of Communication

Journal of Intelligent and Robotic Systems
Multiagent meta-level control for radar coordination

Web Intelligence and Agent Systems
The effectiveness of peer-designed agents in agent-based simulations

Multiagent and Grid Systems

Quantified Score

Hi-index	0.00

Visualization

Abstract

We consider decentralized control of Markov decision processes and give complexity bounds on the worst-case running time for algorithms that find optimal solutions. Generalizations of both the fully observable case and the partially observable case that allow for decentralized control are described. For even two agents, the finite-horizon problems corresponding to both of these models are hard for nondeterministic exponential time. These complexity results illustrate a fundamental difference between centralized and decentralized control of Markov decision processes. In contrast to the problems involving centralized control, the problems we considerprovably do not admit polynomial-time algorithms. Furthermore, assuming EXP ? NEXP, the problems require superexponential time to solve in the worst case.