Decision-Theoretic Planning: Structural Assumptions and Computational Leverage

Abstract

Planning under uncertainty is a central problem in the study of automated sequential decision making, and has been addressed by researchers in many di erent elds, including AI planning, decision analysis, operations research, control theory and economics. While the assumptions and perspectives adopted in these areas often di er in substantial ways, many planning problems of interest to researchers in these elds can be modeled as Markov decision processes (MDPs) and analyzed using the techniques of decision theory.This paper presents an overview and synthesis of MDP-related methods, showing how they provide a unifying framework for modeling many classes of planning problems studied in AI. It also describes structural properties of MDPs that, when exhibited by particular classes of problems, can be exploited in the construction of optimal or approximately optimal policies or plans. Planning problems commonly possess structure in the reward and value functions used to describe performance criteria, in the functions used to describe state transitions and observations, and in the relationships among features used to describe states, actions, rewards, and observations. Specialized representations, and algorithms employing these representations, can achieve computational leverage by exploiting these various forms of structure. Certain AI techniques| in particular those based on the use of structured, intensional representations|can be viewed in this way. This paper surveys several types of representations for both classical and decision-theoretic planning problems, and planning algorithms that exploit these representations in a number of di erent ways to ease the computational burden of constructing policies or plans. It focuses primarily on abstraction, aggregation and decomposition techniques based on AI-style representations.

Computational Decision Support Regret-Based Models for Optimization and Preference Elicitation

“…Markov decision processes (MDPs) are used to model sequential decision problems under uncertainty in economics, operations research, computer science, and many other areas (Puterman, 1994;Boutilier et al, 1999). Unlike the problems above, where a single decision is made, in an MDP one determines a sequence of decisions/actions to guide some stochastic system into certain desirable states.…”

“…Navigating a system in this way requires the appropriate choice of policy π which dictates which action to take at any system state. An optimal policy is one which maximizes the expected sum of rewards accrued, and can be computed using a variety of dynamic and linear programming methods (Puterman, 1994;Boutilier et al, 1999).…”

“…One can distinguish MDPs with unstructured states from those with "factored" state spaces (Boutilier et al, 1999) in which each state is defined as an instantiation of certain variables. In the former case, one can ask queries about the specific reward of a state, e.g., bound queries (Regan and Boutilier, 2009) or precise reward queries (Delage and Mannor, 2007); or one can ask for a comparison of two states (e.g., comparison queries).…”

Reinforcement Learning: A Tutorial Survey and Recent Advances

“…See Chapter 9 (section 10) of Gosavi (2003) for discussions of other model-based RL algorithms. Use of Bayesian networks to represent the TPs in a compact manner was pioneered by Boutilier et al (1999), and was called a factored MDP. The TPMs are learned as functions of a few scalars via a Bayesian network.…”

Solving Hybrid Markov Decision Processes

“…Abstraction and aggregation are techniques [2] that aid factored representations to avoid this problem. Several authors use these notions to find computationally feasible methods for the construction of (approximately) optimal and satisfying policies.…”