Approximate dynamic programming via iterated Bellman inequalities

Abstract

SUMMARYIn this paper we introduce new methods for finding functions that lower bound the value function of a stochastic control problem, using an iterated form of the Bellman inequality. Our method is based on solving linear or semidefinite programs, and produces both a bound on the optimal objective, as well as a suboptimal policy that appears to work very well. These results extend and improve bounds obtained in a previous paper using a single Bellman inequality condition. We describe the methods in a general setting, and show how they can be applied in specific cases including the finite state case, constrained linear quadratic control, switched affine control, and multi-period portfolio investment.

Dynamic Power Management for Portable Hybrid Power-Supply Systems Utilizing Approximate Dynamic Programming

“…This value-function-based method is called DP, and a variety of topics on stochastic optimal control and DP are well-addressed by [5][6][7][8]. A large class of stochastic optimal control problems deal with the dynamics of the form in Equation (1) and are concerned with finding a state-feedback control policy:…”

“…where T is the Bellman operator (see e.g., [5]), whose domain and codomain are both function spaces mapping X onto…”

“…As is well known, the Bellman equation or its operator equation version cannot be solved exactly, except in simple special cases [5,6], and an alternative strategy when finding the exact state value function is the use of ADP relying on an approximate state value function ˆ: V X R  . In the DPM problem discussed in this paper, we are concerned with an ADP solution utilizing convex quadratic functions.…”

“…In the DPM problem discussed in this paper, we are concerned with an ADP solution utilizing convex quadratic functions. This class of quadratic functions is a good choice for the approximate state value function [5,8] because these functions are convenient and powerful for handling various optimization steps appearing in the ADP procedure.…”

“…This value-function-based method is called dynamic programming (DP) and provides an important theoretical foundation for various optimal control problems. However, most real-world optimal control problems cannot be solved by DP exactly, except in very simple cases, and many studies rely on ADP, which finds an approximate solution to the stochastic optimal control problem utilizing an approximate value function, in order to obtain a good suboptimal control policy [5][6][7][8][9]. The main objective of this paper is to consider an ADP-based solution to dynamic power management (DPM) for portable HPSSs employing batteries and supercapacitors.…”