Learning to make predictions in partially observable environments without a generative model
Journal of Artificial Intelligence Research
| Hi-index | 0.00 |
An agent behaving in an unknown environment may wish to learn a model that allows it to make predictions about future events and to anticipate the consequences of its actions. Such a model can greatly enhance the agent's ability to make good decisions. However, in environments like the physical world in which we live, which is stochastic, partially observable, and high dimensional, learning a model is a challenge. One natural approach when faced with a difficult model learning problem is not to model the entire system. Instead, one might focus on capturing the most important aspects of the environment and give up on modeling complicated, irrelevant phenomena. This intuition can be formalized using partial models, which are models that make only a restricted set of (abstract) predictions in only a restricted set of circumstances. Because a partial model has limited prediction responsibilities, it may be significantly simpler than a complete model. Partial models (and similar ideas) have been studied in many contexts, mostly under the Markov assumption, where the agent is assumed to have access to the full state of the world. In this setting, predictions can typically be learned directly as functions of state and the process of learning a partial model is often as simple as estimating only the desired predictions and omitting the rest from the model. As such, much of the relevant work has focused on the interesting and challenging question of which partial models should be learned (rather than how to learn them). In the partially observable case, however, where the state of the world is (more naturally) assumed to be hidden from the agent, just the basic problem of how to learn a partial model poses significant challenges. The goal of this thesis is to provide general results and methods for learning partial models in partially observable systems. Some of the main challenges posed by partial observability are formalized and learning methods are developed to address some of these issues. The learning methods presented are demonstrated empirically to be able to learn partial models in systems that are too complex for standard, complete model learning methods. Finally, many partial models are learned and composed to form complete models that are used for model-based planning in high dimensional arcade game examples.