Provable Reinforcement Learning with a Short-Term Memory

Efroni, Yonathan; Jin, Chi; Krishnamurthy, Akshay; Miryoosefi, Sobhan

doi:10.48550/arxiv.2202.03983

Cited by 6 publications

(26 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, block MDPs satisfy Assumption 1 with α ≥ 1/ √ O. We remark that most existing results for block MDPs or m-step decodable POMDPs (see, e.g., Krishnamurthy et al, 2016;Jiang et al, 2017;Du et al, 2019;Misra et al, 2020;Efroni et al, 2022) further involves decoder class or value function approximation, which is beyond the scope of this paper.…”

Section: Related Workmentioning

confidence: 92%

“…Block MDPs (Krishnamurthy et al, 2016) are POMDPs whose current latent state can be uniquely determined by the current observation. m-step decodable POMDPs (Efroni et al, 2022) are their generalizations whose latent state can be uniquely determined by the most recent history (of observations and actions) of a short length m. Block MDPs and mstep decodable POMDPs are special cases of single-step and m-step weakly revealing POMDPs, respectively. For example, block MDPs satisfy Assumption 1 with α ≥ 1/ √ O.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

When Is Partially Observable Reinforcement Learning Not Scary?

Liu¹,

Chung²,

Szepesvári³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Applications of Reinforcement Learning (RL), in which agents learn to make a sequence of decisions despite lacking complete information about the latent states of the controlled system, that is, they act under partial observability of the states, are ubiquitous. Partially observable RL can be notoriously difficult-well-known information-theoretic results show that learning partially observable Markov decision processes (POMDPs) requires an exponential number of samples in the worst case. Yet, this does not rule out the existence of large subclasses of POMDPs over which learning is tractable.In this paper we identify such a subclass, which we call weakly revealing POMDPs. This family rules out the pathological instances of POMDPs where observations are uninformative to a degree that makes learning hard. We prove that for weakly revealing POMDPs, a simple algorithm combining optimism and Maximum Likelihood Estimation (MLE) is sufficient to guarantee polynomial sample complexity. To the best of our knowledge, this is the first provably sample-efficient result for learning from interactions in overcomplete POMDPs, where the number of latent states can be larger than the number of observations.

show abstract

Section: Related Workmentioning

confidence: 92%

Section: Related Workmentioning

confidence: 99%

When Is Partially Observable Reinforcement Learning Not Scary?

Liu¹,

Chung²,

Szepesvári³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In particular, Azizzadenesheli et al (2016); Guo et al (2016); Jin et al (2020a); Liu et al (2022) consider the tabular POMDPs with (left) invertible emission matrices. Efroni et al (2022) considers the POMDPs where the state is fully determined by the most recent observations of a fixed length. Cayci et al (2022) analyze POMDPs where a finite internal state can approximately determine the state.…”

Section: Introductionmentioning

confidence: 99%

“…Similarly, Cayci et al (2022) consider POMDPs with a finite concentrability coefficient (Munos, 2003;Chen and Jiang, 2019), where the visitation density of an arbitrary policy is close to that of the optimal policy. In contrast, Jin et al (2020a); Efroni et al (2022); Cai et al (2022) consider POMDPs where strategic exploration is necessary.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency

Wang¹,

Cai²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reinforcement learning in partially observed Markov decision processes (POMDPs) faces two challenges. (i) It often takes the full history to predict the future, which induces a sample complexity that scales exponentially with the horizon. (ii) The observation and state spaces are often continuous, which induces a sample complexity that scales exponentially with the extrinsic dimension. Addressing such challenges requires learning a minimal but sufficient representation of the observation and state histories by exploiting the structure of the POMDP.To this end, we propose a reinforcement learning algorithm named Embed to Control (ETC), which learns the representation at two levels while optimizing the policy. (i) For each step, ETC learns to represent the state with a low-dimensional feature, which factorizes the transition kernel. (ii) Across multiple steps, ETC learns to represent the full history with a low-dimensional embedding, which assembles the per-step feature. We integrate (i) and (ii) in a unified framework that allows a variety of estimators (including maximum likelihood estimators and generative adversarial networks). For a class of POMDPs with a low-rank structure in the transition kernel, ETC attains an O(1/ǫ 2 ) sample complexity that scales polynomially with the horizon and the intrinsic dimension (that is, the rank). Here ǫ is the optimality gap. To our best knowledge, ETC is the first sample-efficient algorithm that bridges representation learning and policy optimization in POMDPs with infinite observation and state spaces.

show abstract

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

Liu¹,

Min²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study offline reinforcement learning (RL) in partially observable Markov decision processes. In particular, we aim to learn an optimal policy from a dataset collected by a behavior policy which possibly depends on the latent state. Such a dataset is confounded in the sense that the latent state simultaneously affects the action and the observation, which is prohibitive for existing offline RL algorithms. To this end, we propose the Proxy variable Pessimistic Policy Optimization (P3O) algorithm, which addresses the confounding bias and the distributional shift between the optimal and behavior policies in the context of general function approximation. At the core of P3O is a coupled sequence of pessimistic confidence regions constructed via proximal causal inference, which is formulated as minimax estimation. Under a partial coverage assumption on the confounded dataset, we prove that P3O achieves a n −1/2 -suboptimality, where n is the number of trajectories in the dataset. To our best knowledge, P3O is the first provably efficient offline RL algorithm for POMDPs with a confounded dataset. Contents

show abstract

Provable Reinforcement Learning with a Short-Term Memory

Cited by 6 publications

References 13 publications

When Is Partially Observable Reinforcement Learning Not Scary?

When Is Partially Observable Reinforcement Learning Not Scary?

Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

Contact Info

Product

Resources

About