$\sqrt{n}$-Regret for Learning in Markov Decision Processes with Function Approximation and Low Bellman Rank

Abstract

In this paper, we consider the problem of online learning of Markov decision processes (MDPs) with very large state spaces. Under the assumptions of realizable function approximation and low Bellman ranks, we develop an online learning algorithm that learns the optimal value function while at the same time achieving very low cumulative regret during the learning process. Our learning algorithm, Adaptive Value-function Elimination (AVE), is inspired by the policy elimination algorithm proposed in [1], known as OLIVE. One of our key technical contributions in AVE is to formulate the elimination steps in OLIVE as contextual bandit problems. This technique enables us to apply the active elimination and expert weighting methods from [2], instead of the random action exploration scheme used in the original OLIVE algorithm, for more efficient exploration and better control of the regret incurred in each policy elimination step. To the best of our knowledge, this is the first √ n-regret result for reinforcement learning in stochastic MDPs with general value function approximation.

Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium

“…Most of the existing work focus on the tabular setting; see e.g., Strehl et al (2006); Jaksch et al (2010); Osband et al (2014); Osband and Van Roy (2016); Azar et al (2017); Dann et al (2017); Agrawal and Jia (2017); Jin et al (2018); Russo (2019); Rosenberg and Mansour (2019a,b); Jin and Luo (2019); Zanette and Brunskill (2019); Simchowitz and Jamieson (2019); Dong et al (2019b) and the references therein. Under the function approximation setting, sample-efficient algorithms have been proposed using linear function approximators (Abbasi-Yadkori et al, 2019a,b;Yang and Wang, 2019a;Du et al, 2019b;Cai et al, 2019;Wang et al, 2019), as well as nonlinear ones (Wen and Van Roy, 2017;Jiang et al, 2017;Dann et al, 2018;Du et al, 2019b;Dong et al, 2019a;Du et al, 2019a). Among these results, our work is most related to ; ; Cai et al (2019), which consider linear MDP models and propose optimistic and randomized variants of least-squares value iteration (LSVI) (Bradtke and Barto, 1996;Osband et al, 2014) as well as optimistic variants of proximal policy optimization (Schulman et al, 2017).…”

Provably Efficient Exploration in Policy Optimization

“…Broadly speaking, our work is related to a vast body of work on value-based reinforcement learning in tabular (Jaksch et al, 2010;Osband et al, 2014;Osband and Van Roy, 2016;Azar et al, 2017;Dann et al, 2017;Strehl et al, 2006;Jin et al, 2018) and linear settings (Yang and Wang, 2019a,b;Jin et al, 2019), as well as nonlinear settings involving general function approximators (Wen and Van Roy, 2017;Jiang et al, 2017;Du et al, 2019b;Dong et al, 2019). In particular, our setting is the same as the linear setting studied by Jin et al (2019), which generalizes the one proposed by Yang and Wang (2019a,b).…”

“…In particular, our setting is the same as the linear setting studied by Jin et al (2019), which generalizes the one proposed by Yang and Wang (2019a,b). Also, our setting is a special case of the low-Bellman-rank setting studied by Jiang et al (2017); Dong et al (2019) with Bellman-rank at most d. In comparison, we focus on policy-based reinforcement learning, which is significantly less studied in theory. In particular, compared with optimistic LSVI (Jin et al, 2019), OPPO attains the same regret even in the presence of adversarially chosen reward functions.…”

“…In particular, compared with optimistic LSVI (Jin et al, 2019), OPPO attains the same regret even in the presence of adversarially chosen reward functions. Compared with optimism-led iterative value-function elimination (OLIVE) (Jiang et al, 2017;Dong et al, 2019), which handles the more general low-Bellman-rank setting but is only sample-efficient, OPPO simultaneously attains computational efficiency and sample efficiency in the linear setting. Despite the differences between policy-based and value-based reinforcement learning, our work shows that the general principle of "optimism in the face of uncertainty" (Auer et al, 2002;Bubeck and Cesa-Bianchi, 2012) can be carried over from existing algorithms based on value iteration, e.g., optimistic LSVI, into policy optimization algorithms, e.g., NPG, TRPO, and PPO, to make them sample-efficient, which further leads to a new general principle of "conservative optimism in the face of uncertainty and adversary" that additionally allows adversarially chosen reward functions.…”

FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs

“…There are methods for more general, non-linear, function approximation, but these works either (a) require strong environment assumptions such as determinism (Wen and Van Roy, 2013;, (b) require strong function class assumptions such as bounded Eluder dimension (Russo and Van Roy, 2013;Osband and Van Roy, 2014), (c) have sample complexity scaling linearly with the function class size (Lattimore et al, 2013;Ortner et al, 2014) or (d) are computationally intractable (Jiang et al, 2017;Sun et al, 2019;Dong et al, 2019). Note that Ortner et al (2014); Jiang et al (2015) consider a form of representation learning, abstraction selection, but the former scales linearly with the number of candidate abstractions, while the latter does not address exploration.…”