Mean Field for Markov Decision Processes: From Discrete to Continuous Optimization

Abstract

We study the convergence of Markov Decision Processes made of a large number of objects to optimization problems on ordinary differential equations (ODE). We show that the optimal reward of such a Markov Decision Process, satisfying a Bellman equation, converges to the solution of a continuous Hamilton-Jacobi-Bellman (HJB) equation based on the mean field approximation of the Markov Decision Process. We give bounds on the difference of the rewards, and a constructive algorithm for deriving an approximating solution to the Markov Decision Process from a solution of the HJB equations. We illustrate the method on three examples pertaining respectively to investment strategies, population dynamics control and scheduling in queues are developed. They are used to illustrate and justify the construction of the controlled ODE and to show the gain obtained by solving a continuous HJB equation rather than a large discrete Bellman equation.

Nonlinear Markov Games on a Finite State Space (Mean-field and Binary Interactions)

“…The closest to our setting seems to be the recent work (Gast, Gaujal, & Le Boudec, 2010), which is devoted to a convergence result similar to our Theorem 2. However, in (Gast, Gaujal, & Le Boudec, 2010) a continuous time mean-field control model is obtained as a limit of discrete-time models, and we work directly with continuous time models.…”

“…(Andersson & Djehiche, 2003;Buckdahn, B. Djehiche, J. Li, & S. Peng, 2009) and references therein, for diffusion based models, and (Le Boudec, McDonald, & Mundinger, 2007;Gast & B. Gaujal, 2009;Gast, Gaujal, & Le Boudec, 2010;Bordenave, McDonald, & Proutiere, 2007;Benaïm & Le Boudec, 2008;Milutinovic & Lima, 2006) for discrete models, more engineering application oriented. In these papers one can find various concrete applications (from robot swamps to transportation theory and networks), which are also relevant to the mathematical models discussed in the present paper.…”

“…However, in (Gast, Gaujal, & Le Boudec, 2010) a continuous time mean-field control model is obtained as a limit of discrete-time models, and we work directly with continuous time models. More essentially, under a slightly stronger regularity assumptions on the model (basically continuous differentiability of coefficients instead of Lipschitz continuity) we obtain explicit rate of convergence for the averages over empirical measures, and in some cases even for adaptive control policies.…”

A Limit Theorem for Markov Decision Processes

“…Therefore this paper is closer in spirit to stochastic approximation theory (Benaïm, 1998). While writing this paper I have learned from the paper by Gast et al (2012). They establish a limit result for a finite-horizon Markov decision process converging to a deterministic optimal control problem.…”

A limit theorem for Markov decision processes

“…First I study infinite horizon problems with discounting. Second, my proof techniques are based on dynamic programming and viscosity solution techniques, whereas Gast et al (2012) rely on ideas from stochastic approximation theory. Before developing the general analysis of the problem, let me introduce some concrete examples to which the limit results apply.…”