Nearly Optimal Policy Optimization with Stable at Any Time Guarantee

Abstract

Policy optimization methods are one of the most widely used classes of Reinforcement Learning (RL) algorithms. However, theoretical understanding of these methods remains insufficient. Even in the episodic (time-inhomogeneous) tabular setting, the state-of-the-art theoretical result of policy-based method in Shani et al. ( 2020) is only O( √ S 2 AH 4 K) where S is the number of states, A is the number of actions, H is the horizon, and K is the number of episodes, and there is a √ SH gap compared with the information theoretic lower bound Ω( √ SAH 3 K) (Jin et al., 2018). To bridge such a gap, we propose a novel algorithm Reference-based Policy Optimization with Stable at Any Time guarantee (RPO-SAT), which features the property "Stable at Any Time". We prove that our algorithm achieves O( √ SAH 3 K + √ AH 4 K) regret. When S > H, our algorithm is minimax optimal when ignoring logarithmic factors. To our best knowledge, RPO-SAT is the first computationally efficient, nearly minimax optimal policy-based algorithm for tabular RL.

Policy Optimization for Stochastic Shortest Path

“…This cost is often of order Õ( √ K) and is one of the dominating terms in the regret bound; see e.g. (Shani et al, 2020;Wu et al, 2021) for the finite-horizon case. For SSP, this is undesirable because it also depends on T ⋆ or even T max .…”

Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game

“…Variance reduction via reference function originates from the optimization literature [Johnson and Zhang, 2013]. Recently, several RL works [Zhang et al, 2020, Wu et al, 2021, Xie et al, 2021a, Cui and Du, 2022 also adopt this technique to obtain sharper bounds. Nevertheless, these works only focus on the tabular setting, and the linear case is more complicated and thus requires refined analysis.…”