Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature

Dong, Kefan; Yang, Jiaqi; Ma, Tengyu

doi:10.48550/arxiv.2102.04168

Cited by 7 publications

(13 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Going beyond linear function approximation, systematic exploration strategies are developed based on structural assumptions on MDP such as low Bellman rank (Jiang et al, 2017) and block MDP (Du et al, 2019). These methods are either computationally intractable (Jiang et al, 2017;Sun et al, 2019;Ayoub et al, 2020;Zanette et al, 2020;Dong et al, 2021; or are only oracle efficient (Feng et al, 2020;Agarwal et al, 2020b). The recent work Feng et al (2021) provides a sample efficient approach with non-linear policies, however, the algorithm requires maintaining the functional form of all prior policies.…”

Section: Related Workmentioning

confidence: 99%

MADE: Exploration via Maximizing Deviation from Explored Regions

Zhang¹,

Rashidinejad²,

Jiao³

et al. 2021

Preprint

View full text Add to dashboard Cite

In online reinforcement learning (RL), efficient exploration remains particularly challenging in high-dimensional environments with sparse rewards. In low-dimensional environments, where tabular parameterization is possible, count-based upper confidence bound (UCB) exploration methods achieve minimax near-optimal rates. However, it remains unclear how to efficiently implement UCB in realistic RL tasks that involve non-linear function approximation. To address this, we propose a new exploration approach via maximizing the deviation of the occupancy of the next policy from the explored regions. We add this term as an adaptive regularizer to the standard RL objective to balance exploration vs. exploitation. We pair the new objective with a provably convergent algorithm, giving rise to a new intrinsic reward that adjusts existing bonuses. The proposed intrinsic reward is easy to implement and combine with other existing RL algorithms to conduct exploration. As a proof of concept, we evaluate the new intrinsic reward on tabular examples across a variety of model-based and model-free algorithms, showing improvements over count-only exploration strategies. When tested on navigation and locomotion tasks from MiniGrid and DeepMind Control Suite benchmarks, our approach significantly improves sample efficiency over state-of-the-art methods. Our code is available at https://github.com/tianjunz/MADE.

show abstract

Section: Related Workmentioning

confidence: 99%

MADE: Exploration via Maximizing Deviation from Explored Regions

Zhang¹,

Rashidinejad²,

Jiao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…They also showed that the eluder dimension is small in several settings, including generalized linear models and LQR. However, as shown in [Dong et al, 2021], the eluder dimension could be exponentially large even with a single ReLU neuron, which suggested the eluder dimension would face difficulty in dealing with neural network cases. The eluder dimension is only known to give non-trivial bounds for linear function classes and monotone functions of linear function classes.…”

Section: Related Workmentioning

confidence: 99%

“…These results rely on finite state space or exact linear approximations. Recently, sample efficient algorithms under non-linear function approximation settings are proposed [Wen and Van Roy, 2017, Dann et al, 2018, Du et al, 2019b, Dong et al, 2020, Wang et al, 2020a, Dong et al, 2021. Those algorithms are based on Bellman rank [Jiang et al, 2017], eluder dimension [Russo and Van Roy, 2013b], neural tangent kernel [Jacot et al, 2018, Du et al, 2019a, or sequential Rademacher complexity [Rakhlin et al, 2015a,b].…”

Section: Introductionmentioning

confidence: 99%

“…Those algorithms are based on Bellman rank [Jiang et al, 2017], eluder dimension [Russo and Van Roy, 2013b], neural tangent kernel [Jacot et al, 2018, Du et al, 2019a, or sequential Rademacher complexity [Rakhlin et al, 2015a,b]. Besides, general neural network function approximation is shown to be hard by Dong et al [2021]. Yet, there is a mismatch between the empirical success of deep RL and the theoretical understanding of RL under general deep neural network function approximations, which yields the following important question:…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Going Beyond Linear RL: Sample Efficient Neural Function Approximation

Huang¹,

Huang²,

Kakade³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep Reinforcement Learning (RL) powered by neural net approximation of the Q function has had enormous empirical success. While the theory of RL has traditionally focused on linear function approximation (or eluder dimension) approaches, little is known about nonlinear RL with neural net approximations of the Q functions. This is the focus of this work, where we study function approximation with two-layer neural networks (considering both ReLU and polynomial activation functions). Our first result is a computationally and statistically efficient algorithm in the generative model setting under completeness for two-layer neural networks. Our second result considers this setting but under only realizability of the neural net function class. Here, assuming deterministic dynamics, the sample complexity scales linearly in the algebraic dimension. In all cases, our results significantly improve upon what can be attained with linear (or eluder dimension) methods.

show abstract

“…Our lower bound construction is inspired by the similar construction in Theorem 5.1 of Dong et al (2021).…”

Section: Diversity Of Non-linear Function Classesmentioning

confidence: 99%

Representation Learning Beyond Linear Prediction Functions

Xu¹,

Tewari²

2021

Preprint

View full text Add to dashboard Cite

Recent papers on the theory of representation learning has shown the importance of a quantity called diversity when generalizing from a set of source tasks to a target task. Most of these papers assume that the function mapping shared representations to predictions is linear, for both source and target tasks. In practice, researchers in deep learning use different numbers of extra layers following the pretrained model based on the difficulty of the new task. This motivates us to ask whether diversity can be achieved when source tasks and the target task use different prediction function spaces beyond linear functions. We show that diversity holds even if the target task uses a neural network with multiple layers, as long as source tasks use linear functions. If source tasks use nonlinear prediction functions, we provide a negative result by showing that depth-1 neural networks with ReLu activation function need exponentially many source tasks to achieve diversity. For a general function class, we find that eluder dimension gives a lower bound on the number of tasks required for diversity. Our theoretical results imply that simpler tasks generalize better. Though our theoretical results are shown for the global minimizer of empirical risks, their qualitative predictions still hold true for gradient-based optimization algorithms as verified by our simulations on deep neural networks.

show abstract

Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvature

Cited by 7 publications

References 12 publications

MADE: Exploration via Maximizing Deviation from Explored Regions

MADE: Exploration via Maximizing Deviation from Explored Regions

Going Beyond Linear RL: Sample Efficient Neural Function Approximation

Representation Learning Beyond Linear Prediction Functions

Contact Info

Product

Resources

About