A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning

Ren, Tongzheng; Zhang, Tianjun; Szepesvári, Csaba; Dai, Bo

doi:10.48550/arxiv.2111.11485

Cited by 3 publications

(7 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 1. To our knowledge, only (Ren et al 2021) and this work provide examples of linear transitions in RL with continuous state-actions. The former consider Gaussian transitions with unknown mean (f ⋆ (s, a)) and known variance, i.e.…”

Section: Bilinear Exponential Family Of Mdpsmentioning

confidence: 99%

“…Thus, different works (Shariff and Szepesvári 2020;Lattimore, Szepesvari, and Weisz 2020;Van Roy and Dong 2019) propose to leverage different low-dimensional representations of value functions or transitions to perform efficient planning. Here, we take note from (Ren et al 2021) that Gaussian transitions induce an explicit linear value function in an RKHS. And generalize this observation with the bilinear exponential.…”

Section: Related Work: Functional Representations Of Mdps With Regret...mentioning

confidence: 99%

“…We illustrate this comparison in Table ??. We highlight three technical tools that we used to improve the previous analyses: 1) Using transportation inequalities instead of the simulation lemma reduces a √ H factor compared to (Ren et al 2021), 2) Leveraging the observation that true value functions are bounded enables using an improved elliptical lemma (compared to (Chowdhury, Gopalan, and Maillard 2021)), and 3) Noticing that the norm of features can only be large for a finite amount of time allows us to forgo clipping and reduce a √ d factor from the regret compared to (Zanette et al 2020).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration & Planning

Ouhamma

Basu

Maillard

2023

AAAI

View full text Add to dashboard Cite

We study the problem of episodic reinforcement learning in continuous state-action spaces with unknown rewards and transitions. Specifically, we consider the setting where the rewards and transitions are modeled using parametric bilinear exponential families. We propose an algorithm, that a) uses penalized maximum likelihood estimators to learn the unknown parameters, b) injects a calibrated Gaussian noise in the parameter of rewards to ensure exploration, and c) leverages linearity of the bilinear exponential family transitions with respect to an underlying RKHS to perform tractable planning. We provide a frequentist regret upper-bound for our algorithm which, in the case of tabular MDPs, is order-optimal with respect to H and K, where H is the episode length and K is the number of episodes. Our analysis improves the existing bounds for the bilinear exponential family of MDPs by square root of H and removes the handcrafted clipping deployed in existing RLSVI-type algorithms.

show abstract

Section: Bilinear Exponential Family Of Mdpsmentioning

confidence: 99%

Section: Related Work: Functional Representations Of Mdps With Regret...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration & Planning

Ouhamma

Basu

Maillard

2023

AAAI

View full text Add to dashboard Cite

show abstract

“…Low-rank MDPs Low-rank MDP is strictly more general than linear MDPs which assume representation is known a priori. There are several related papers come from the recent literature on provable representation learning for low-rank MDPs (Agarwal et al, 2020b;Modi et al, 2021;Uehara et al, 2021;Ren et al, 2021). Low-rank MDPs generalize Block MDPs, so these algorithms are applicable in our setting.…”

Section: Rl With Function Approximationmentioning

confidence: 99%

Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach

Zhang¹,

Song²,

Uehara³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present BRIEE (Block-structured Representation learning with Interleaved Explore Exploit), an algorithm for efficient reinforcement learning in Markov Decision Processes with block structured dynamics (i.e., Block MDPs), where rich observations are generated from a set of unknown latent states. BRIEE interleaves latent states discovery, exploration, and exploitation together, and can provably learn a near-optimal policy with sample complexity scaling polynomially in the number of latent states, actions, and the time horizon, with no dependence on the size of the potentially infinite observation space. Empirically, we show that BRIEE is more sample efficient than the state-of-art Block MDP algorithm HOMER and other empirical RL baselines on challenging rich-observation combination lock problems which require deep exploration. *

show abstract

“…Several related works studied low-rank MDPs with provable sample complexities. [Agarwal et al, 2020b, Ren et al, 2021 and [Uehara et al, 2021] consider the model-based setting, where the algorithm learns the representation with the model class of the transition probability given. [Modi et al, 2021] provided a representation learning algorithm under the model-free setting and proved its sample efficiency when the MDP satisfies the minimal reachability assumption.…”

Section: Related Workmentioning

confidence: 99%

Representation Learning for General-sum Low-rank Markov Games

Ni¹,

Song²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study multi-agent general-sum Markov games with nonlinear function approximation. We focus on low-rank Markov games whose transition matrix admits a hidden low-rank structure on top of an unknown non-linear representation. The goal is to design an algorithm that (1) finds an ε-equilibrium policy sample efficiently without prior knowledge of the environment or the representation, and (2) permits a deeplearning friendly implementation. We leverage representation learning and present a model-based and a model-free approach to construct an effective representation from the collected data. For both approaches, the algorithm achieves a sample complexity of poly(H, d, A, 1/ε), where H is the game horizon, d is the dimension of the feature vector, A is the size of the joint action space and ε is the optimality gap. When the number of players is large, the above sample complexity can scale exponentially with the number of players in the worst case. To address this challenge, we consider Markov games with a factorized transition structure and present an algorithm that escapes such exponential scaling. To our best knowledge, this is the first sample-efficient algorithm for multi-agent general-sum Markov games that incorporates (non-linear) function approximation. We accompany our theoretical result with a neural network-based implementation of our algorithm and evaluate it against the widely used deep RL baseline, DQN with fictitious play.

show abstract

A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning

Cited by 3 publications

References 32 publications

Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration & Planning

Bilinear Exponential Family of MDPs: Frequentist Regret Bound with Tractable Exploration & Planning

Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach

Representation Learning for General-sum Low-rank Markov Games

Contact Info

Product

Resources

About