Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs

Zhong, Han; Yang, Zhuoran; Wang, Zhaoran; Szepesvári, Csaba

doi:10.48550/arxiv.2110.08984

Cited by 3 publications

(8 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the variation budget is known a prior, (Fei et al 2020) propose the first policy-based method for non-stationary RL, but they assume stationary transitions and adversarial fullinformation rewards in the tabular setting. (Zhong, Yang, and Szepesvári 2021) extends the above results to a more general setting where both the transitions and rewards can vary over episodes. To eliminate the assumption of having prior knowledge on variation budgets, (Wei and Luo 2021) recently outline that an adaptive restart approach can be used to convert any upper-confidence-bound-type stationary RL algorithm to a dynamic-regret-minimizing algorithm.…”

Section: Related Worksupporting

confidence: 62%

“…Non-stationary RL. Non-stationary RL has been mostly studied in the unconstrained setting (Jaksch, Ortner, and Auer 2010;Auer, Gajane, and Ortner 2019;Ortner, Gajane, and Auer 2020;Domingues et al 2021;Mao et al 2020;Zhou et al 2020;Touati and Vincent 2020;Fei et al 2020;Zhong, Yang, and Szepesvári 2021;Cheung, Simchi-Levi, and Zhu 2020;Wei and Luo 2021). Our work is related to policy-based methods for non-stationary RL since the optimal solution of CMDP is usually a stochastic policy (Altman 1999) and thus a policy-based method is preferred.…”

Section: Related Workmentioning

confidence: 99%

“…The notion of B ⋆ is also used for online convex optimization with a dynamic regret criterion (Besbes, Gur, and Zeevi 2015;Willett 2013, 2015;Cao and Liu 2018) and for policy-based methods in non-stationary unconstrained MDPs (Fei et al 2020;Zhong, Yang, and Szepesvári 2021). It is worth noting that the variations (B P , B ◇ ) and B ⋆ do not imply each other.…”

Section: Assumptionmentioning

confidence: 99%

See 2 more Smart Citations

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding

Lavaei

2023

AAAI

View full text Add to dashboard Cite

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

show abstract

Section: Related Worksupporting

confidence: 62%

Section: Related Workmentioning

confidence: 99%

Section: Assumptionmentioning

confidence: 99%

See 1 more Smart Citation

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Ding

Lavaei

2023

AAAI

View full text Add to dashboard Cite

show abstract

“…Our work contributes to the theoretical investigations of policy-based methods in RL (Cai et al, 2020;Shani et al, 2020;Lancewicki et al, 2020;Fei et al, 2020;He et al, 2021;Zhong et al, 2021;Luo et al, 2021;Zanette et al, 2021). The most related policy-based method is proposed by Shani et al (2020), who also studies the episodic tabular MDPs with unknown transitions, stochastic losses, and bandit feedback.…”

Section: Related Workmentioning

confidence: 88%

Nearly Optimal Policy Optimization with Stable at Any Time Guarantee

Wu¹,

Yang²,

Han³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Policy optimization methods are one of the most widely used classes of Reinforcement Learning (RL) algorithms. However, theoretical understanding of these methods remains insufficient. Even in the episodic (time-inhomogeneous) tabular setting, the state-of-the-art theoretical result of policy-based method in Shani et al. ( 2020) is only O( √ S 2 AH 4 K) where S is the number of states, A is the number of actions, H is the horizon, and K is the number of episodes, and there is a √ SH gap compared with the information theoretic lower bound Ω( √ SAH 3 K) (Jin et al., 2018). To bridge such a gap, we propose a novel algorithm Reference-based Policy Optimization with Stable at Any Time guarantee (RPO-SAT), which features the property "Stable at Any Time". We prove that our algorithm achieves O( √ SAH 3 K + √ AH 4 K) regret. When S > H, our algorithm is minimax optimal when ignoring logarithmic factors. To our best knowledge, RPO-SAT is the first computationally efficient, nearly minimax optimal policy-based algorithm for tabular RL.

show abstract

“…Non-stationary RL has been mostly studied in the risk-neutral setting. When the variation budget is known a prior, a common strategy for adapting to the non-stationarity is to follow the forgetting principle, such as the restart strategy (Mao et al 2020;Zhou et al 2020;Zhao et al 2020;Ding and Lavaei 2022), exponential decayed weights (Touati and Vincent 2020), or sliding window (Cheung, Simchi-Levi, and Zhu 2020;Zhong, Yang, and Szepesvári 2021). In this work, we focus on the restart method mainly due to its advantage of the simplicity of the the memory efficiency (Zhao et al 2020) and generalize it to the risk-sensitive RL setting.…”

Section: Related Workmentioning

confidence: 99%

Non-stationary Risk-Sensitive Reinforcement Learning: Near-Optimal Dynamic Regret, Adaptive Detection, and Separation Design

Ding

Jin

Lavaei

2023

AAAI

View full text Add to dashboard Cite

We study risk-sensitive reinforcement learning (RL) based on an entropic risk measure in episodic non-stationary Markov decision processes (MDPs). Both the reward functions and the state transition kernels are unknown and allowed to vary arbitrarily over time with a budget on their cumulative variations. When this variation budget is known a prior, we propose two restart-based algorithms, namely Restart-RSMB and Restart-RSQ, and establish their dynamic regrets. Based on these results, we further present a meta-algorithm that does not require any prior knowledge of the variation budget and can adaptively detect the non-stationarity on the exponential value functions. A dynamic regret lower bound is then established for non-stationary risk-sensitive RL to certify the near-optimality of the proposed algorithms. Our results also show that the risk control and the handling of the non-stationarity can be separately designed in the algorithm if the variation budget is known a prior, while the non-stationary detection mechanism in the adaptive algorithm depends on the risk parameter. This work offers the first non-asymptotic theoretical analyses for the non-stationary risk-sensitive RL in the literature.

show abstract

Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs

Cited by 3 publications

References 45 publications

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Nearly Optimal Policy Optimization with Stable at Any Time Guarantee

Non-stationary Risk-Sensitive Reinforcement Learning: Near-Optimal Dynamic Regret, Adaptive Detection, and Separation Design

Contact Info

Product

Resources

About