Learning in Observable POMDPs, without Computationally Intractable Oracles

Abstract

Much of reinforcement learning theory is built on top of oracles that are computationally hard to implement. Specifically for learning near-optimal policies in Partially Observable Markov Decision Processes (POMDPs), existing algorithms either need to make strong assumptions about the model dynamics (e.g. deterministic transitions) or assume access to an oracle for solving a hard optimistic planning or estimation problem as a subroutine. In this work we develop the first oracle-free learning algorithm for POMDPs under reasonable assumptions. Specifically, we give a quasipolynomial-time end-to-end algorithm for learning in "observable" POMDPs, where observability is the assumption that well-separated distributions over states induce well-separated distributions over observations. Our techniques circumvent the more traditional approach of using the principle of optimism under uncertainty to promote exploration, and instead give a novel application of barycentric spanners to constructing policy covers.

Reward-Mixing MDPs with a Few Latent Contexts are Learnable

“…As a special case of POMDPs, we may consider applying the POMDP solutions for learning a near-optimal policy in RMMDPs. There is a growing body of work that focuses on the case when single or multiple-step observations from test action sequences are sufficient statistics of the environment (e.g., [4,28,2,19,14,34,46]). In such a scenario, latent model parameters can be learned up to some parameter transformations when the system is irreducible or optimistically explored.…”

“…proposed in [34,19]). The work in [30] does not give a satisfactory solution either since it requires a similar assumption of strong separability between latent contexts.…”