An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

Wang, Yuanhao; Wang, Ruosong; Kakade, Sham M.

doi:10.48550/arxiv.2103.12690

Cited by 10 publications

(11 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A subject of much recent interest in the RL community is that of RL with function approximation. The majority of attention has been devoted to MDPs with linear structure Jin et al, 2020b;Wang et al, 2019;Du et al, 2019;Zanette et al, 2020a,b;Ayoub et al, 2020;Jia et al, 2020;Weisz et al, 2021;Zhou et al, 2020Zhou et al, , 2021Zhang et al, 2021b;Wang et al, 2021;Wagenmaker et al, 2021a). Several different settings of MDPs with linear structure have been proposed.…”

Section: Related Workmentioning

confidence: 99%

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

Wagenmaker¹,

Chen²,

Simchowitz³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reward-free reinforcement learning (RL) considers the setting where the agent does not have access to a reward function during exploration, but must propose a near-optimal policy for an arbitrary reward function revealed only after exploring. In the the tabular setting, it is well known that this is a more difficult problem than PAC RL-where the agent has access to the reward function during exploration-with optimal sample complexities in the two settings differing by a factor of |S|, the size of the state space. We show that this separation does not exist in the setting of linear MDPs. We first develop a computationally efficient algorithm for reward-free RL in a d-dimensional linear MDP with sample complexity scaling as O(d 2 / 2 ). We then show a matching lower bound of Ω(d 2 / 2 ) on PAC RL. To our knowledge, our approach is the first computationally efficient algorithm to achieve optimal d dependence in linear MDPs, even in the single-reward PAC setting. Our algorithm relies on a novel procedure which efficiently traverses a linear MDP, collecting samples in any given "feature direction", and enjoys a sample complexity scaling optimally in the (linear MDP equivalent of the) maximal state visitation probability. We show that this exploration procedure can also be applied to solve the problem of obtaining "well-conditioned" covariates in linear MDPs.

show abstract

Section: Related Workmentioning

confidence: 99%

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

Wagenmaker¹,

Chen²,

Simchowitz³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Several recent works have extended their results significantly (Du et al, 2021;Jin et al, 2021). In the special case of linear function approximation, a vast body of recent work exists Jin et al, 2020b;Wang et al, 2019;Du et al, 2019;Zanette et al, 2020a,b;Ayoub et al, 2020;Jia et al, 2020;Weisz et al, 2021;Zhou et al, 2020Zhou et al, , 2021Zhang et al, 2021;Wang et al, 2021). A variety of assumptions are made in these works, and we highlight two of them in particular.…”

Section: Related Workmentioning

confidence: 94%

First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approach

Wagenmaker¹,

Chen²,

Simchowitz³

et al. 2021

Preprint

View full text Add to dashboard Cite

Obtaining first-order regret bounds-regret bounds scaling not as the worst-case but with some measure of the performance of the optimal policy on a given instance-is a core question in sequential decision-making. While such bounds exist in many settings, they have proven elusive in reinforcement learning with large state spaces. In this work we address this gap, and show that it is possible to obtain regret scaling as O( V 1 K) in reinforcement learning with large state spaces, namely the linear MDP setting. Here V 1 is the value of the optimal policy and K is the number of episodes. We demonstrate that existing techniques based on least squares estimation are insufficient to obtain this result, and instead develop a novel robust self-normalized concentration bound based on the robust Catoni mean estimator, which may be of independent interest.

show abstract

“…Concentrability places stronger restrictions on the data distribution and underlying dynamics, and always implies identifiability when the state and action space are finite. Further work in this direction includes (1) Zanette (2021), who provides a slightly more general lower bound for linear realizability, and (2) lower bounds for online reinforcement learning with linear realizability (Du et al, 2020;Weisz et al, 2021;Wang et al, 2021b). It is worth noting that Zanette (2021) provides a lower bound with a policy-induced data distribution, where over-coverage cannot occur; however concentrability is not satisfied by his construction.…”

Section: Related Workmentioning

confidence: 99%

Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation

Foster¹,

Krishnamurthy²,

Simchi‐Levi³

et al. 2021

Preprint

View full text Add to dashboard Cite

We consider the offline reinforcement learning problem, where the aim is to learn a decision making policy from logged data. Offline RL-particularly when coupled with (value) function approximation to allow for generalization in large or continuous state spaces-is becoming increasingly relevant in practice, because it avoids costly and time-consuming online data collection and is well suited to safety-critical domains. Existing sample complexity guarantees for offline value function approximation methods typically require both (1) distributional assumptions (i.e., good coverage) and (2) representational assumptions (i.e., ability to represent some or all Q-value functions) stronger than what is required for supervised learning. However, the necessity of these conditions and the fundamental limits of offline RL are not well understood in spite of decades of research. This led Chen and Jiang (2019) to conjecture that concentrability (the most standard notion of coverage) and realizability (the weakest representation condition) alone are not sufficient for sample-efficient offline RL. We resolve this conjecture in the positive by proving that in general, even if both concentrability and realizability are satisfied, any algorithm requires sample complexity polynomial in the size of the state space to learn a non-trivial policy.Our results show that sample-efficient offline reinforcement learning requires either restrictive coverage conditions or representation conditions that go beyond supervised learning, and highlight a phenomenon called over-coverage which serves as a fundamental barrier for offline value function approximation methods. A consequence of our results for reinforcement learning with linear function approximation is that the separation between online and offline RL can be arbitrarily large, even in constant dimension.

show abstract

An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

Cited by 10 publications

References 17 publications

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processes

First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approach

Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation

Contact Info

Product

Resources

About