Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs

Liu, Tao; Zhou, Ruida; Kalathil, Dileep; Tian, Chao

doi:10.48550/arxiv.2106.02684

Cited by 4 publications

(18 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The study of RL algorithms for CMDPs has received considerable attention due to the safety requirement (Altman, 1999;Paternain et al, 2019;Yu et al, 2019;Dulac-Arnold et al, 2019;Garcıa & Fernández, 2015). Our work is closely related to Lagrangian-based CMDP algorithms with optimistic policy evaluations (Efroni et al, 2020;Singh et al, 2020;Ding et al, 2021;Liu et al, 2021;Qiu et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…Ding et al (2021) generalize the above results to the linear kernel CMDPs. Under some mild conditions and additional computation cost, Liu et al (2021) propose two algorithms to learn policies with a zero or bounded constraint violation for CMDPs. Beyond the stationary CMDP, Qiu et al (2020) consider the online CMDPs where only the rewards in objective can vary over episodes.…”

Section: Related Workmentioning

confidence: 99%

“…The feasibility assumption has often been used for the stationary CMDPs (Ding et al, 2021;Ying et al, 2021;Liu et al, 2021). Under this assumption, it is convenient to establish the strong duality and the boundedness of the optimal dual variable.…”

Section: Let the Policy Spacementioning

confidence: 99%

Section: Let the Policy Spacementioning

confidence: 99%

“…This assumption adapts the definition of linear kernel MDP (Ayoub et al, 2020;Cai et al, 2020;Zhou et al, 2021) to CMDP and has also been used in (Ding et al, 2021) for stationary constrained MDP problems. Some examples of linear kernel MDPs include tabular MDPs (Zhou et al, 2021), feature embedded transition models , and linear combinations of base models (Modi et al, 2020). The linear kernal MDP defined in Assumption 2.3 is different from linear MDP Jin et al, 2020) since they define transition dynamics using the different feature maps, although both of them encapsulate the tabular MDP as the special case.…”

Section: Let the Policy Spacementioning

confidence: 99%

See 4 more Smart Citations

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding¹,

Lavaei²

2022

Preprint

View full text Add to dashboard Cite

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which play a central role in ensuring the safety of RL in timevarying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we propose a Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that features three mechanisms: periodic-restart-based policy improvement, dual update with dual regularization, and periodicrestart-based optimistic policy evaluation. We establish a dynamic regret bound and a constraint violation bound for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Let the Policy Spacementioning

confidence: 99%

Section: Let the Policy Spacementioning

confidence: 99%

Section: Let the Policy Spacementioning

confidence: 99%

See 3 more Smart Citations

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding¹,

Lavaei²

2022

Preprint

View full text Add to dashboard Cite

show abstract

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding

Lavaei

2023

AAAI

View full text Add to dashboard Cite

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

show abstract

On Kernelized Multi-Armed Bandits with Constraints

Zhou¹,

Jiang²

2022

Preprint

View full text Add to dashboard Cite

We study a stochastic bandit problem with a general unknown reward function and a general unknown constraint function. Both functions can be non-linear (even non-convex) and are assumed to lie in a reproducing kernel Hilbert space (RKHS) with a bounded norm. This kernelized bandit setup strictly generalizes standard multi-armed bandits and linear bandits. In contrast to safety-type hard constraints studied in prior works, we consider soft constraints that may be violated in any round as long as the cumulative violations are small, which is motivated by various practical applications. Our ultimate goal is to study how to utilize the nature of soft constraints to attain a finer complexity-regret-constraint trade-off in the kernelized bandit setting. To this end, leveraging primal-dual optimization, we propose a general framework for both algorithm design and performance analysis. This framework builds upon a novel sufficient condition, which not only is satisfied under general exploration strategies, including upper confidence bound (UCB), Thompson sampling (TS), and new ones based on random exploration, but also enables a unified analysis for showing both sublinear regret and sublinear or even zero constraint violation. We demonstrate the superior performance of our proposed algorithms via numerical experiments based on both synthetic and real-world datasets. Along the way, we also make the first detailed comparison between two popular methods for analyzing constrained bandits and Markov decision processes (MDPs) by discussing the key difference and some subtleties in the analysis, which could be of independent interest to the communities.

show abstract

Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs

Cited by 4 publications

References 13 publications

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

On Kernelized Multi-Armed Bandits with Constraints

Contact Info

Product

Resources

About