Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime

Abstract

We study the global convergence of policy gradient for infinite-horizon, continuous state and action space, entropy-regularized Markov decision processes (MDPs). We consider a softmax policy with (onehidden layer) neural network approximation in a mean-field regime. Additional entropic regularization in the associated mean-field probability measure is added, and the corresponding gradient flow is studied in the 2-Wasserstein metric. We show that the objective function is increasing along the gradient flow. Further, we prove that if the regularization in terms of the mean-field measure is sufficient, the gradient flow converges exponentially fast to the unique stationary solution, which is the unique maximizer of the regularized MDP objective. Lastly, we study the sensitivity of the value function along the gradient flow with respect to regularization parameters and the initial condition. Our results rely on the careful analysis of non-linear Fokker-Planck-Kolmogorov equation and extend the pioneering work of [MXSS20] and [AKLM20], which quantify the global convergence rate of policy gradient for entropy-regularized MDPs in the tabular setting.

Linear convergence of a policy gradient method for finite horizon continuous time stochastic control problems

“…Conditions (ii)-(iv) help to ease the nonconvexity of φ → J(α φ ; ξ 0 ) and to reduce the oscillation of the loss function's curvature, which subsequently promotes the convergence of gradient-based algorithms (see [29]). Condition (ii), along with Example 2.3, also justifies recent reinforcement learning heuristics that adding f-divergences, such as the relative entropy, to the optimization objective can accelerate the convergence of PGMs (see e.g., [34,19]).…”

“…As alluded to earlier, the map V A ∋ φ → J(α φ ; ξ 0 ) ∈ R ∪ {∞} is typically nonconvex and may not satisfy the Polyak-Lojasiewicz condition as in the setting with parametric policies ( [10,38,26,11,13,19]). Hence to ensure the linear convergence of the PPGM (1.7), we impose further conditions on the coefficients which guarantee that we are in one of the following five cases:…”

“…Most existing theoretical results of PGMs, especially those establishing (optimal) linear convergence, focus on discrete time problems and restrict policies within specific parametric families. This includes Markov decision problems (MDPs) with softmax parameterized policies [26] or overparametrized one-hidden-layer neural-network policies [38,11,19], and discrete time linear (LQ) control problems with linear parameterized policies [10,13]. The analysis therein exploits heavily the specific structure of the considered (discrete time) control problems and policy parameterizations, and hence is difficult to extend to general continuous time control problems or general policy parameterizations.…”

Convergence of policy gradient methods for finite-horizon stochastic linear-quadratic control problems

“…These algorithms parametrise the policy as a function of the system state, and update the policy parametrisation based on the gradient of the control objective. Most of the progress, especially the convergence analysis of PG methods, has been in discrete-time Markov decision processes (MDPs) (see e.g., [5,10,18,34,16]). However, most real-world control systems, such as those in aerospace, the automotive industry and robotics, are naturally continuous-time dynamical systems, and hence do not fit in the MDP setting.…”