Ordinary Differential Equation Methods for Markov Decision Processes and Application to Kullback--Leibler Control Cost

Abstract

A new approach to computation of optimal policies for MDP (Markov decision process) models is introduced. The main idea is to solve not one, but an entire family of MDPs, parameterized by a scalar ζ that appears in the one-step reward function. For an MDP with d states, the family of value functions {h * ζ : ζ ∈ R} is the solution to an ODE,where the vector field V : R d → R d has a simple form, based on a matrix inverse. This general methodology is applied to a family of average-cost optimal control models in which the one-step reward function is defined by Kullback-Leibler divergence. The motivation for this reward function in prior work is computation: The solution to the MDP can be expressed in terms of the Perron-Frobenius eigenvector for an associated positive matrix. The drawback with this approach is that no hard constraints on the control are permitted. It is shown here that it is possible to extend this framework to model randomness from nature that cannot be modified by the controller. Perron-Frobenius theory is no longer applicable -the resulting dynamic programming equations appear as complex as a completely unstructured MDP model. Despite this apparent complexity, it is shown that this class of MDPs admits a solution via this ODE technique. This approach is new and practical even for the simpler problem in which randomness from nature is absent.

Distributed Control Design for Balancing the Grid Using Flexible Loads

“…For more history the reader is referred to [36,14], in addition to the papers surveyed in Section 3.2. While beyond the scope of this article, it is important to note that Todorov's 'linearly solvable' MDP model [44] is similar to prior work such as [24], and the form of the solution could have been anticipated from well-known results in the theory of large-deviations for Markov chains [7]. It is pointed out in [45] that this approach has a long history in the context of controlled stochastic differential equations [18].…”

“…Consider a load model in which the full state space is the Cartesian product X = X u × X n , where X u are components of the state that can be directly manipulated through control. In prior work [7,6], the following conditional-independence structure is assumed: for each state x = (x u , x n ), and each ζ ∈ R,…”

“…A computational ODE approach is introduced in [7,6]: for a vector field V whose domain and range are functions on X × X u ,…”

“…The function H 0 = V (h 0 ) can be used in the exponential family design (9). It is shown in [7] that this function is a solution to Poisson's equation for the nominal model:…”

Distributed Control of Thermostatically Controlled Loads: Kullback-Leibler Optimal Control in Continuous Time

“…The main contribution of this paper is to demonstrate that the local control designs introduced in [3], [4], [21] admit practical extension to continuous-state models. This is true even in the significantly more complex setting in which the nominal dynamics include stochastic disturbances that are outside of direct control (such as variations in ambient temperature, or inlet temperature of water to the TCL).…”