Gap-Dependent Unsupervised Exploration for Reinforcement Learning

Wu, Jingfeng; Braverman, Vladimir; Yang, Lin F.

doi:10.48550/arxiv.2108.05439

Cited by 2 publications

(1 citation statement)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reward-free RL setting seeks to explore an MDP without access to a reward function, in order to then determine a near-optimal policy for an arbitrary reward function. This has been studied in the tabular setting (Jin et al, 2020a;Ménard et al, 2020;Zhang et al, 2020a;Wu et al, 2021), where the optimal scaling is known to be Θ(|S| 2 |A|/ 2 ) (Jin et al, 2020a), as well as the function approximation setting (Zanette et al, 2020c;Wang et al, 2020;Zhang et al, 2021a). In the linear MDP setting, Wang et al (2020) show a sample complexity of O( d 3 H 6 2 ), and Zanette et al (2020c) show a complexity of…”

Section: Related Workmentioning

confidence: 99%

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

Wagenmaker¹,

Chen²,

Simchowitz³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reward-free reinforcement learning (RL) considers the setting where the agent does not have access to a reward function during exploration, but must propose a near-optimal policy for an arbitrary reward function revealed only after exploring. In the the tabular setting, it is well known that this is a more difficult problem than PAC RL-where the agent has access to the reward function during exploration-with optimal sample complexities in the two settings differing by a factor of |S|, the size of the state space. We show that this separation does not exist in the setting of linear MDPs. We first develop a computationally efficient algorithm for reward-free RL in a d-dimensional linear MDP with sample complexity scaling as O(d 2 / 2 ). We then show a matching lower bound of Ω(d 2 / 2 ) on PAC RL. To our knowledge, our approach is the first computationally efficient algorithm to achieve optimal d dependence in linear MDPs, even in the single-reward PAC setting. Our algorithm relies on a novel procedure which efficiently traverses a linear MDP, collecting samples in any given "feature direction", and enjoys a sample complexity scaling optimally in the (linear MDP equivalent of the) maximal state visitation probability. We show that this exploration procedure can also be applied to solve the problem of obtaining "well-conditioned" covariates in linear MDPs.

show abstract

Section: Related Workmentioning

confidence: 99%

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

Wagenmaker¹,

Chen²,

Simchowitz³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver

Chen¹,

Hu²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although model-based reinforcement learning (RL) approaches are considered more sample efficient, existing algorithms are usually relying on sophisticated planning algorithm to couple tightly with the model-learning procedure. Hence the learned models may lack the ability of being re-used with more specialized planners. In this paper we address this issue and provide approaches to learn an RL model efficiently without the guidance of a reward signal. In particular, we take a plug-in solver approach, where we focus on learning a model in the exploration phase and demand that any planning algorithm on the learned model can give a near-optimal policy. Specicially, we focus on the linear mixture MDP setting, where the probability transition matrix is a (unknown) convex combination of a set of existing models. We show that, by establishing a novel exploration algorithm, the plug-in approach learns a model by taking Õ(d 2 H 3 /ǫ 2 ) interactions with the environment and any ǫ-optimal planner on the model gives an O(ǫ)-optimal policy on the original model. This sample complexity matches lower bounds for non-plug-in approaches and is statistically optimal. We achieve this result by leveraging a careful maximum total-variance bound using Bernstein inequality and properties specified to linear mixture MDP.

show abstract

Gap-Dependent Unsupervised Exploration for Reinforcement Learning

Cited by 2 publications

References 20 publications

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver

Contact Info

Product

Resources

About