Efficient Planning in Large MDPs with Weak Linear Function Approximation

Shariff, Roshan; Szepesvári, Csaba

doi:10.48550/arxiv.2007.06184

Cited by 5 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the sample complexity in Du et al [2019c] has at least linear dependency on the number of actions, whereas our sample complexity in Section 5 has no dependency on the size of the action space. Finally, Shariff and Szepesvári [2020] obtains a polynomial upper bound under the realizability assumption when the features for all state-action pairs are inside the convex hull of a polynomial-sized coreset and the generative model is available to the agent.…”

Section: Related Workmentioning

confidence: 99%

An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

Wang¹,

Wang²,

Kakade³

2021

Preprint

View full text Add to dashboard Cite

A fundamental question in the theory of reinforcement learning is: suppose the optimal Q-function lies in the linear span of a given d dimensional feature mapping, is sample-efficient reinforcement learning (RL) possible? The recent and remarkable result of resolved this question in the negative, providing an exponential (in d) sample size lower bound, which holds even if the agent has access to a generative model of the environment. One may hope that this information theoretic barrier for RL can be circumvented by further supposing an even more favorable assumption: there exists a constant suboptimality gap between the optimal Q-value of the best action and that of the second-best action (for all states). The hope is that having a large suboptimality gap would permit easier identification of optimal actions themselves, thus making the problem tractable; indeed, provided the agent has access to a generative model, sample-efficient RL is in fact possible with the addition of this more favorable assumption.This work focuses on this question in the standard online reinforcement learning setting, where our main result resolves this question in the negative: our hardness result shows that an exponential sample complexity lower bound still holds even if a constant suboptimality gap is assumed in addition to having a linearly realizable optimal Qfunction. Perhaps surprisingly, this implies an exponential separation between the online RL setting and the generative model setting. Complementing our negative hardness result, we give two positive results showing that provably sample-efficient RL is possible either under an additional low-variance assumption or under a novel hypercontractivity assumption (both implicitly place stronger conditions on the underlying dynamics model).

show abstract

Section: Related Workmentioning

confidence: 99%

An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

Wang¹,

Wang²,

Kakade³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, Du et al (2020a); examined how the model misspecification error propagates and impacts the sample efficiency of policy learning. A related line of works assumed that the linear function class is closed or has low approximation error under the Bellman operator, referred to as low inherent Bellman error (Munos, 2005;Shariff and Szepesvári, 2020;Zanette et al, 2019Zanette et al, , 2020b.…”

Section: Additional Related Workmentioning

confidence: 99%

Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting

Li¹,

Chen²,

Chi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Low-complexity models such as linear function representation play a pivotal role in enabling sampleefficient reinforcement learning (RL). The current paper pertains to a scenario with value-based linear representation, which postulates the linear realizability of the optimal Q-function (also called the "linear Q problem"). While linear realizability alone does not allow for sample-efficient solutions in general, the presence of a large sub-optimality gap is a potential game changer, depending on the sampling mechanism in use. Informally, sample efficiency is achievable with a large sub-optimality gap when a generative model is available, but is unfortunately infeasible when we turn to standard online RL settings.In this paper, we make progress towards understanding this linear Q problem by investigating a new sampling protocol, which draws samples in an online/exploratory fashion but allows one to backtrack and revisit previous states in a controlled and infrequent manner. This protocol is more flexible than the standard online RL setting, while being practically relevant and far more restrictive than the generative model. We develop an algorithm tailored to this setting, achieving a sample complexity that scales polynomially with the feature dimension, the horizon, and the inverse sub-optimality gap, but not the size of the state/action space. Our findings underscore the fundamental interplay between sampling protocols and low-complexity structural representation in RL.

show abstract

“…Since these works need to find a globally optimal design or barycentric spanner, their computational complexities depend polynomially on the size of the state space. Under the V * -realizability assumption (i.e., the optimal value function is linear in some feature map), Shariff and Szepesvári [2020] proposed a planning algorithm assuming the availability Table 1: Recent advances on RL algorithms with linear function approximation under different assumptions. Positive results mean query complexity depends only polynomially on the relative parameter while negative results refer an exponential lower bound on the query complexity.…”

Section: Related Workmentioning

confidence: 99%

“…†: The algorithms in these works are not query or computationally efficient unless the agent is provided with an approximate optimal design [Lattimore et al, 2020] or barycentric spanner [Du et al, 2020] or "core states" [Shariff and Szepesvári, 2020] Q * -realizability, constant gap N/A online access of a set of core states but obtaining such core states can still be computationally inefficient. Zanette et al [2019] proposed an algorithm that uses a similar concept named anchor points but only provided a greedy heuristic to generate these points.…”

Section: Related Workmentioning

confidence: 99%

“…For a tuple z, we use z s , z a , z φ , and z q to denote the s, a, φ, and q coordinates of z, respectively. We note that in prior works, the core set usually consists of the state-action pairs and their features [Lattimore et al, 2020, Du et al, 2020, Shariff and Szepesvári, 2020; whereas in this paper, for the convenience of notation, we also have the target values (Q-function estimates) in the core set elements.…”

Section: Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Local Planning with Linear Function Approximation

Yin¹,

Hao²,

Abbasi-Yadkori³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We study query and computationally efficient planning algorithms with linear function approximation and a simulator. We assume that the agent only has local access to the simulator, meaning that the agent can only query the simulator at states that have been visited before. This setting is more practical than many prior works on reinforcement learning with a generative model. We propose an algorithm named confident Monte Carlo least square policy iteration (CONFIDENT MC-LSPI) for this setting. Under the assumption that the Q-functions of all deterministic policies are linear in known features of the state-action pairs, we show that our algorithm has polynomial query and computational complexities in the dimension of the features, the effective planning horizon and the targeted sub-optimality, while these complexities are independent of the size of the state space. One technical contribution of our work is the introduction of a novel proof technique that makes use of a virtual policy iteration algorithm. We use this method to leverage existing results on ℓ ∞ -bounded approximate policy iteration to show that our algorithm can learn the optimal policy for the given initial state even only with local access to the simulator. We believe that this technique can be extended to broader settings beyond this work.

show abstract

Efficient Planning in Large MDPs with Weak Linear Function Approximation

Cited by 5 publications

References 24 publications

An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap

Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting

Efficient Local Planning with Linear Function Approximation

Contact Info

Product

Resources

About