Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximation

Abstract

Policy optimization methods are popular reinforcement learning algorithms, because their incremental and on-policy nature makes them more stable than the value-based counterparts. However, the same properties also make them slow to converge and sample inefficient, as the on-policy requirement precludes data reuse and the incremental updates couple large iteration complexity into the sample complexity. These characteristics have been observed in experiments as well as in theory in the recent work of Agarwal et al. (2020a), which provides a policy optimization method PC-PG that can robustly find near optimal polices for approximately linear Markov decision processes but suffers from an extremely poor sample complexity compared with value-based techniques.In this paper, we propose a new algorithm, COPOE, that overcomes the sample complexity issue of PC-PG while retaining its robustness to model misspecification. Compared with PC-PG, COPOE makes several important algorithmic enhancements, such as enabling data reuse, and uses more refined analysis techniques, which we expect to be more broadly applicable to designing new reinforcement learning algorithms. The result is an improvement in sample complexity from O(1/ǫ 11 ) for PC-PG to O(1/ǫ 3 ) for COPOE, nearly bridging the gap with value-based techniques.

Nearly Optimal Policy Optimization with Stable at Any Time Guarantee

“…Our work contributes to the theoretical investigations of policy-based methods in RL (Cai et al, 2020;Shani et al, 2020;Lancewicki et al, 2020;Fei et al, 2020;He et al, 2021;Zhong et al, 2021;Luo et al, 2021;Zanette et al, 2021). The most related policy-based method is proposed by Shani et al (2020), who also studies the episodic tabular MDPs with unknown transitions, stochastic losses, and bandit feedback.…”

Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonuses

“…To address this issue, we adopt the idea of policy cover, recently introduced in [Agarwal et al, 2020a, Zanette et al, 2021. Specifically, we spend the first T 0 rounds to find an exploratory (mixture) policy π cov (called policy cover) which tends to reach all possible directions of the feature space.…”

“…Motivated by this issue, a line of recent works [Cai et al, 2020, Shani et al, 2020, Agarwal et al, 2020a, Zanette et al, 2021 equip policy optimization with global exploration by adding exploration bonuses to the update, and prove favorable guarantees even without making extra exploratory assumptions. Moreover, they all demonstrate some robustness aspect of policy optimization (such as being able to handle adversarial losses or a certain degree of model misspecification).…”

“…Moreover, they all demonstrate some robustness aspect of policy optimization (such as being able to handle adversarial losses or a certain degree of model misspecification). Despite these important progresses, however, many limitations still exist, including worse regret rates comparing to the best value-based or model-based approaches [Shani et al, 2020, Agarwal et al, 2020a, Zanette et al, 2021, or requiring full-information feedback on the entire loss function (as opposed to the more realistic bandit feedback) [Cai et al, 2020].…”

“…Most of them study stochastic/stationary losses, with the exception of [Cai et al, 2020, Neu and Olkhovskaya, 2020. Our algorithm for the linear MDP setting bears some similarity to those of [Agarwal et al, 2020a, Zanette et al, 2021 which consider stationary losses. However, in each episode, their algorithms first execute an exploratory policy (from a policy cover), and then switch to the policy suggested by the policy optimization algorithm, which inevitably leads to linear regret when facing adversarial losses.…”