Exponential filter stability via Dobrushin’s coefficient

“…Examples of the latter type include algorithms employing various pruning heuristics [Mon82, CLZ97, HF00a, Hau00, RG02, TK03, PB04, SV05, PGT06, RPPCD08, SV10, SS12, SYHL13, GHL19] and algorithms which optimize over restricted classes of policies [Han98, MKKC99, KMN99, LYX11, AYA18]. To our knowledge, some of the only works presenting subexponential time approximate planning algorithms are [BDRS96] and [MY20,KY20] (ignoring end-to-end learning algorithms, which we discuss later).…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, this property plays a key role in applications, in terms of bounding the effect of misspecification and analyzing the asymptotic [MS92, SM94] and non-asymptotic [CKMY21] robustness. For finite, unstructured HMMs (our setting), there are asymptotic stability results under analogues of observability [VH09a], and exponential stability results under mixing assumptions [SAD98,BK98,MY20]. In this context, Theorem 1.3 is the first exponential stability result for unstructured HMMs without mixing assumptions.…”

Section: Introductionmentioning

confidence: 99%

Planning in Observable POMDPs in Quasipolynomial Time

Golowich¹,

Moitra²,

Rohatgi³

2022

Preprint

View full text Add to dashboard Cite

Partially Observable Markov Decision Processes (POMDPs) are a natural and general model in reinforcement learning that take into account the agent's uncertainty about its current state. In the literature on POMDPs, it is customary to assume access to a planning oracle that computes an optimal policy when the parameters are known, even though the problem is known to be computationally hard. Almost all existing planning algorithms either run in exponential time, lack provable performance guarantees, or require placing strong assumptions on the transition dynamics under every possible policy. In this work, we revisit the planning problem and ask: Are there natural and well-motivated assumptions that make planning easy?Our main result is a quasipolynomial-time algorithm for planning in (one-step) observable POMDPs. Specifically, we assume that well-separated distributions on states lead to wellseparated distributions on observations, and thus the observations are at least somewhat informative in each step. Crucially, this assumption places no restrictions on the transition dynamics of the POMDP; nevertheless, it implies that near-optimal policies admit quasi-succinct descriptions, which is not true in general (under standard hardness assumptions). Our analysis is based on new quantitative bounds for filter stability -i.e. the rate at which an optimal filter for the latent state forgets its initialization. Furthermore, we prove matching hardness for planning in observable POMDPs under the Exponential Time Hypothesis.

show abstract

“…U n which are obtained under π * . Proposition 1 implies that, if the underlying Markov chain is ergodic in the sense of Condition 2, different priors are forgotten at a geometric rate in n, similar to the case of HMCs [39,38,40]. In the case of finite-state POMDPs, for tabular Q-learning, a different characterization of the inference error was presented in [25] under an assumption on the Dobrushin coefficient.…”

Section: Condition 2 (Minorization-majorization)mentioning

confidence: 99%

Learning to Control Partially Observed Systems with Finite Memory

Cayci¹,

He²,

Srikant³

2022

Preprint

View full text Add to dashboard Cite

We consider the reinforcement learning problem for partially observed Markov decision processes (POMDPs) with large or even countably infinite state spaces, where the controller has access to only noisy observations of the underlying controlled Markov chain. We consider a natural actor-critic method that employs a finite internal memory for policy parameterization, and a multi-step temporal difference learning algorithm for policy evaluation. We establish, to the best of our knowledge, the first non-asymptotic global convergence of actor-critic methods for partially observed systems under function approximation. In particular, in addition to the function approximation and statistical errors that also arise in MDPs, we explicitly characterize the error due to the use of finite-state controllers. This additional error is stated in terms of the total variation distance between the traditional belief state in POMDPs and the posterior distribution of the hidden state when using a finite-state controller. Further, we show that this error can be made small in the case of sliding-block controllers by using larger block sizes.

show abstract

Exponential filter stability via Dobrushin’s coefficient

Cited by 19 publications

References 14 publications

Robustness to Approximations and Model Learning in MDPs and POMDPs

Robustness to Approximations and Model Learning in MDPs and POMDPs

Planning in Observable POMDPs in Quasipolynomial Time

Learning to Control Partially Observed Systems with Finite Memory

Contact Info

Product

Resources

About