Approximate Dynamic Programming

Abstract

This is my preface. I am going to explain why I wrote this book and who it is for. Chapter 1The challenges of dynamic programmingThe optimization of problems over time arises in many settings, ranging from the control of heating systems to managing entire economies. In between are examples including landing aircraft, purchasing new equipment, managing blood inventories, scheduling fleets of vehicles, selling assets, investing money in portfolios or just playing a game of tic-tac-toe or backgammon. These problems involve making decisions, then observing information, after which we make more decisions, and then more information, and so on. Known as sequential decision problems, they can be straightforward (if subtle) to formulate, but solving them is another matter.Dynamic programming has its roots in several fields. Engineering and economics tend to focus on problems with continuous states and decisions (these communities refer to decisions as controls), while the fields of operations research and artificial intelligence work primarily with discrete states and decisions (or actions). Problems that are modeled with continuous states and decisions (and typically in continuous time) are often addressed under the umbrella of "control theory" whereas problems with discrete states and decisions, modeled in discrete time, are studied at length under the umbrella of "Markov decision processes." Both of these subfields set up recursive equations that depend on the use of a state variable to capture history in a compact way. There are many high-dimensional problems such as those involving the allocation of resources that are generally studied using the tools of mathematical programming. Most of this work focuses on deterministic problems using tools such as linear, nonlinear or integer programming, but there is a subfield known as stochastic programming which incorporates uncertainty. Our presentation spans all of these fields.1 CHAPTER 1. THE CHALLENGES OF DYNAMIC PROGRAMMING

Predicting climate effects on Pacific sardine

“…This type of scenario exploration would tell managers the permissible level of fishing given a target biomass and recent ocean conditions. This approach is analogous to the greedy heuristic strategies that are used in high-dimensional approximate optimization problems (26). Of course, considerable work remains to be done to develop and evaluate management plans based on these methods.…”

Biasing Monte-Carlo Simulations through RAVE Values

“…This algorithm is particularly interesting when the number of states is huge. In this case, classical algorithms like Minimax and Alphabeta [9], for two-player games, and Dynamic Programming [13], for one-player games, are too time-consuming or not efficient. MCTS combines an exploration of the tree based on a compromise between exploration and exploitation, and an evaluation based on Monte-Carlo simulations.…”

Optimized look‐ahead tree policies: a bridge between look‐ahead tree policies and direct policy search

“…It is weaker because it does not have to assign the precise quantity "discounted future sum of rewards"; any number will do as long as it helps to grow the tree in roughly the right direction. It is for this reason that we believe it could be advantageous to try to learn a good scoring function instead of trying to directly learn/approximate the optimal value function: the former can be a rather simple function (as evidenced in the good results we get in Section 5 where we use for all domains exactly the same simple weighted sum of features), whereas the latter would require a far more expressive parametrization (and it is well-known that value function approximation scales badly when the dimensionality of the state space grows [41]). Figure 2 presents a simple algorithm based on a sorted list to implement policies as parameterized look-ahead trees.…”