Achieving Zero Constraint Violation for Concave Utility Constrained Reinforcement Learning via Primal-Dual Approach

Bai, Qinbo; Bedi, Amrit Singh; Agarwal, Mridul; Koppel, Alec; Aggarwal, Vaneet

doi:10.48550/arxiv.2109.06332

Cited by 6 publications

(14 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model-free RL algorithms have also been proposed [8][9][10] to solve CMDP. However, all of the above require a generator model, which simulates from any state and action.…”

Section: Can We Achieve Provably Sample-efficient and Model-free Expl...mentioning

confidence: 99%

“…These issues are exacerbated in the large state space. Hence, there are several recent works starting to investigate model-free algorithms for CMDPs, which directly update the value function or the policy without first estimating the model [8][9][10]. However, all of these works consider an easier setting compared to standard RL in that they assume access to a simulator [11] (a.k.a.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Provably Efficient Model-Free Constrained RL with Linear Function Approximation

Ghosh¹,

Zhou²,

Shroff³

2022

Preprint

View full text Add to dashboard Cite

We study the constrained reinforcement learning problem, in which an agent aims to maximize the expected cumulative reward subject to a constraint on the expected total value of a utility function. In contrast to existing model-based approaches or model-free methods accompanied with a 'simulator', we aim to develop the first model-free, simulator-free algorithm that achieves a sublinear regret and a sublinear constraint violation even in large-scale systems. To this end, we consider the episodic constrained Markov decision processes with linear function approximation, where the transition dynamics and the reward function can be represented as a linear function of some known feature mapping. We show that Õ( √ d 3 H 3 T ) regret and Õ( √ d 3 H 3 T ) constraint violation bounds can be achieved, where d is the dimension of the feature mapping, H is the length of the episode, and T is the total number of steps. Our bounds are attained without explicitly estimating the unknown transition model or requiring a simulator, and they depend on the state space only through the dimension of the feature mapping. Hence our bounds hold even when the number of states goes to infinity. Our main results are achieved via novel adaptations of the standard LSVI-UCB algorithms. In particular, we first introduce primal-dual optimization into the LSVI-UCB algorithm to balance between regret and constraint violation. More importantly, we replace the standard greedy selection with respect to the state-action function in LSVI-UCB with a soft-max policy. This turns out to be key in establishing uniform concentration for the constrained case via its approximation-smoothness trade-off. Finally, we also show that one can achieve an even zero constraint violation for large enough T by trading the regret a little bit but still maintaining the same order with respect to T .

show abstract

“…Model-free RL algorithms have also been proposed [8][9][10] to solve CMDP. However, all of the above require a generator model, which simulates from any state and action.…”

Section: Can We Achieve Provably Sample-efficient and Model-free Expl...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Provably Efficient Model-Free Constrained RL with Linear Function Approximation

Ghosh¹,

Zhou²,

Shroff³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…All rights reserved. 1 The detailed proof can be found in Appendix in (Bai et al 2021) 2012), etc.). The standard MDP equipped with the cost function for the constraints is called constrained Markov Decision process (CMDP) framework (Altman 1999).…”

Section: Introductionmentioning

confidence: 99%

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach

Bai

Bedi

Agarwal

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Reinforcement learning is widely used in applications where one needs to perform sequential decisions while interacting with the environment. The problem becomes more challenging when the decision requirement includes satisfying some safety constraints. The problem is mathematically formulated as constrained Markov decision process (CMDP). In the literature, various algorithms are available to solve CMDP problems in a model-free manner to achieve epsilon-optimal cumulative reward with epsilon feasible policies. An epsilon-feasible policy implies that it suffers from constraint violation. An important question here is whether we can achieve epsilon-optimal cumulative reward with zero constraint violations or not. To achieve that, we advocate the use of a randomized primal-dual approach to solve the CMDP problems and propose a conservative stochastic primal-dual algorithm (CSPDA) which is shown to exhibit O(1/epsilon^2) sample complexity to achieve epsilon-optimal cumulative reward with zero constraint violations. In the prior works, the best available sample complexity for the epsilon-optimal policy with zero constraint violation is O(1/epsilon^5). Hence, the proposed algorithm provides a significant improvement compared to the state of the art.

show abstract

“…Moreover, [36] considered the Nash equilibrium computation for general-sum stochastic Nash games. [37] designed a primal-dual algorithm for constrained markov decision process (equivalent to a saddle point problem) and utilized regret analysis to prove zero constraint violation. In a different line of research, [38] considered the case when a network of cooperative agents need to solve a zero-sum game and proposed a distributed Bregman-divergence algorithm to compute a NE.…”

Section: Related Workmentioning

confidence: 99%

No-Regret Learning in Network Stochastic Zero-Sum Games

Huang¹,

Lei²,

Ye³

2022

Preprint

View full text Add to dashboard Cite

No-regret learning has been widely used to compute a Nash equilibrium in two-person zerosum games. However, there is still a lack of regret analysis for network stochastic zero-sum games, where players competing in two subnetworks only have access to some local information, and the cost functions include uncertainty. Such a game model can be found in security games, when a group of inspectors work together to detect a group of evaders. In this paper, we propose a distributed stochastic mirror descent (D-SMD) method, and establish the regret bounds O( √ T ) and O(log T ) in the expected sense for convex-concave and strongly convexstrongly concave costs, respectively. Our bounds match those of the best known first-order online optimization algorithms. We then prove the convergence of the time-averaged iterates of D-SMD to the set of Nash equilibria. Finally, we show that the actual iterates of D-SMD almost surely converge to the Nash equilibrium in the strictly convex-strictly concave setting.

show abstract

Achieving Zero Constraint Violation for Concave Utility Constrained Reinforcement Learning via Primal-Dual Approach

Cited by 6 publications

References 9 publications

Provably Efficient Model-Free Constrained RL with Linear Function Approximation

Provably Efficient Model-Free Constrained RL with Linear Function Approximation

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach

No-Regret Learning in Network Stochastic Zero-Sum Games

Contact Info

Product

Resources

About