Control of Random Sequences in Problems with Constraints

Piunovskii, A. B.

doi:10.1137/1138077

Cited by 7 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We consider an infinite-horizon discounted constrained Markov decision process [79,6,4] -CMDP ( S, A, P, r, u, b, γ, ρ ) -where S, A are the action/action spaces, P is the transition kernel that specifies the transition probability P (s ′ | s, a) from state s to next state s ′ under action a ∈ A, r, u : S × A → [0, 1] are the reward/utility functions, b is the constraint threshold, γ ∈ [0, 1) is the discount factor, and ρ is the initial state distribution. A stationary stochastic policy π : S → ∆(A) determines a probability distribution ∆(A) over the action space A based on current state, i.e., a t ∼ π(• | s t ) at time t. Let Π be the set of all possible stochastic policies.…”

Section: Preliminariesmentioning

confidence: 99%

“…Constrained Markov decision process (constrained MDP) is the classical formulation for constrained dynamic systems in the early stochastic control literature (e.g., [14,83,79,43,6]) and the recent constrained reinforcement learning (RL) literature (e.g., [18,2,4,91,41,72]). It is applicable to many constrained control problems by integrating other system specifications in constraints, and admits a natural extension of constrained optimization and Lagrangian over policies.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Policy gradient primal-dual mirror descent for constrained MDPs with large state spaces

Ding

Jovanović

2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

We study the problem of computing an optimal policy of an infinite-horizon discounted constrained Markov decision process (constrained MDP). Despite the popularity of Lagrangian-based policy search methods used in practice, the oscillation of policy iterates in these methods has not been fully understood, bringing out issues such as violation of constraints and sensitivity to hyper-parameters. To fill this gap, we employ the Lagrangian method to cast a constrained MDP into a constrained saddle-point problem in which max/min players correspond to primal/dual variables, respectively, and develop two single-time-scale policy-based primal-dual algorithms with non-asymptotic convergence of their policy iterates to an optimal constrained policy. Specifically, we first propose a regularized policy gradient primal-dual (RPG-PD) method that updates the policy using an entropy-regularized policy gradient, and the dual via a quadratic-regularized gradient ascent, simultaneously. We prove that the policy primal-dual iterates of RPG-PD converge to a regularized saddle point with a sublinear rate, while the policy iterates converge sublinearly to an optimal constrained policy. We further instantiate RPG-PD in large state or action spaces by including function approximation in policy parametrization, and establish similar sublinear last-iterate policy convergence. Second, we propose an optimistic policy gradient primal-dual (OPG-PD) method that employs the optimistic gradient method to update primal/dual variables, simultaneously. We prove that the policy primal-dual iterates of OPG-PD converge to a saddle point that contains an optimal constrained policy, with a linear rate. To the best of our knowledge, this work appears to be the first non-asymptotic policy last-iterate convergence result for single-time-scale algorithms in constrained MDPs. We further exhibit the merits and effectiveness of our methods in computational experiments.

show abstract

Section: Preliminariesmentioning

confidence: 99%