Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonuses

Abstract

Policy optimization is a widely-used method in reinforcement learning. Due to its local-search nature, however, theoretical guarantees on global optimality often rely on extra assumptions on the Markov Decision Processes (MDPs) that bypass the challenge of global exploration. To eliminate the need of such assumptions, in this work, we develop a general solution that adds dilated bonuses to the policy update to facilitate global exploration. To showcase the power and generality of this technique, we apply it to several episodic MDP settings with adversarial losses and bandit feedback, improving and generalizing the state-of-the-art. Specifically, in the tabular case, we obtain O( √ T ) regret where T is the number of episodes, improving the O(T 2/3 ) regret bound by Shani et al. [2020]. When the number of states is infinite, under the assumption that the state-action values are linear in some low-dimensional features, we obtain O(T 2/3 ) regret with the help of a simulator, matching the result of Neu and Olkhovskaya [2020] while importantly removing the need of an exploratory policy that their algorithm requires. When a simulator is unavailable, we further consider a linear MDP setting and obtain O(T 14/15 ) regret, which is the first result for linear MDPs with adversarial losses and bandit feedback. * Equal contribution.It can also be defined via the Bellman equation involving the state value function V π (x; ℓ) and the state-action value function Q π (x, a; ℓ) (a.k.a. Q-function) defined as below: V (x H ; ℓ) = 0, Q π (x, a; ℓ) = ℓ(x, a) + E x ′ ∼P (•|x,a) [V π (x ′ ; ℓ)] , and V π (x; ℓ) = E a∼π(•|x) [Q π (x, a; ℓ)] .

Nearly Optimal Policy Optimization with Stable at Any Time Guarantee

“…Our work contributes to the theoretical investigations of policy-based methods in RL (Cai et al, 2020;Shani et al, 2020;Lancewicki et al, 2020;Fei et al, 2020;He et al, 2021;Zhong et al, 2021;Luo et al, 2021;Zanette et al, 2021). The most related policy-based method is proposed by Shani et al (2020), who also studies the episodic tabular MDPs with unknown transitions, stochastic losses, and bandit feedback.…”

Refined Regret for Adversarial MDPs with Linear Function Approximation

“…Lemma 2.5 (Lemma 3.1 by Luo et al (2021a)). If {b k } K k=1 are non-negative, B k (s, a) is defined as in Eq.…”

“…Later, Luo et al (2021b) study bandit-feedback linear-Q MDPs with a simulator, providing an O(d 2/3 H 2 K 2/3 ) regret bound. A refined version (Luo et al, 2021a) considers simulator-free bandit-feedback linear MDPs, giving an O(d 2 H 4 K 14/15 ) bound. Meanwhile, provided with a good exploratory policy (see Footnote 4), these bounds improve to O(poly(d, H) K/λ 0 ) and O(poly(d, H)λ −4/7 0 K 6/7 ), respectively.…”

“…for s ∈ S h (which again uses the simulator; see Algorithm 4 as well as detailed discussions in (Luo et al, 2021a)).…”

“…At last, we also apply our method to simulator-free linear MDPs (formally defined in Theorem 2.4), yielding an efficient algorithm with O(K 8/9 ) regret and greatly outperforming the best existing bound O(K 14/15 ) (Luo et al, 2021a). 3 Remarkably, in this application, we not only use our refined analysis for FTRL with the log-barrier regularizer, but also develop a more sample-efficient alternative to the Matrix Geometric Resampling (MGR) method introduced by Neu and Olkhovskaya (2020) and later adopted by Neu and Olkhovskaya (2021) and Luo et al (2021a), which could also be of independent interest.…”