Online, Informative MCMC Thinning with Kernelized Stein Discrepancy

Hawkins, Cole; Koppel, Alec; Zhang, Zheng

doi:10.48550/arxiv.2201.07130

Cited by 2 publications

(7 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Eqn (22) clearly differentiates our method from [26] highlighting the novelty of our approach is deciphering the application of KSD to our model based RL problem. From the application of Theorem 5 [26] for our formulation with H new samples in the dictionary.…”

Section: B Proof Of Lemma 41mentioning

confidence: 99%

“…This means the posterior defined by compressed dictionary φ D k+1 is at most k+1 in KSD from its uncompressed counterpart. See [26] for related development of this compression routine in the context of MCMC. We will see in the regret analysis section (cf.…”

Section: Kernel Stein Discrepancy and Posterior Coreset Constructionmentioning

confidence: 99%

“…Before starting the proof, here we discuss what is unique about it as compared to the existing literature. The kernel Stein discrepancy based compression exists in the literature [26] but that is limited to the settings of estimating the distributions. This is the first time we are extending the analysis to model based RL settings.…”

Section: B Proof Of Lemma 41mentioning

confidence: 99%

“…( 2)) for each episode. The analysis here follows a similar structure to the one performed in the proof of [26,Theorem 1] with careful adjustments to for the RL setting we are dealing with in this work. Let us now begin the proof.…”

Section: B Proof Of Lemma 41mentioning

confidence: 99%

“…for any arbitrary constant r k > 0. We use the upper bound in (26) to the right hand side of ( 25), to obtain…”

Section: Infmentioning

confidence: 99%

See 4 more Smart Citations

Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning

Chakraborty¹,

Bedi²,

Koppel³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recent progress on improving theoretical sample efficiency of model-based reinforcement learning (RL), which exhibits superior sample complexity in practice, requires Gaussian and Lipschitz assumptions on the transition model, and additionally defines a posterior representation that grows unbounded with time. In this work, we propose a novel Kernelized Stein Discrepancy-based Posterior Sampling for RL algorithm (named KSRL) which extends model-based RL based upon posterior sampling (PSRL) in several ways: we (i) relax the need for any smoothness or Gaussian assumptions, allowing for complex mixture models; (ii) ensure it is applicable to large-scale training by incorporating a compression step such that the posterior consists of a Bayesian coreset of only statistically significant past state-action pairs; and (iii) develop a novel regret analysis of PSRL based upon integral probability metrics, which, under a smoothness condition on the constructed posterior, can be evaluated in closed form as the kernelized Stein discrepancy (KSD). Consequently, we are able to improve the O(H 3/2 d √ T ) regret of PSRL to O(H 3/2 √ T ), where d is the input dimension, H is the episode length, and T is the total number of episodes experienced, alleviating a linear dependence on d . Moreover, we theoretically establish a trade-off between regret rate with posterior representational complexity via introducing a compression budget parameter based on KSD, and establish a lower bound on the required complexity for consistency of the model. Experimentally, we observe that this approach is competitive with several state of the art RL methodologies, with substantive improvements in computation time. Experimentally, we observe that this approach is competitive with several state of the art RL methodologies, and can achieve up-to 50% reduction in wall clock time in some continuous control environments.

show abstract

Section: B Proof Of Lemma 41mentioning

confidence: 99%

Section: Kernel Stein Discrepancy and Posterior Coreset Constructionmentioning

confidence: 99%

Section: B Proof Of Lemma 41mentioning

confidence: 99%

Section: B Proof Of Lemma 41mentioning

confidence: 99%

“…for any arbitrary constant r k > 0. We use the upper bound in (26) to the right hand side of ( 25), to obtain…”

Section: Infmentioning

confidence: 99%

See 3 more Smart Citations

Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning

Chakraborty¹,

Bedi²,

Koppel³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning

Chakraborty

Bedi

Tokekar

et al. 2023

AAAI

View full text Add to dashboard Cite

Model-based approaches to reinforcement learning (MBRL) exhibit favorable performance in practice, but their theoretical guarantees in large spaces are mostly restricted to the setting when transition model is Gaussian or Lipschitz, and demands a posterior estimate whose representational complexity grows unbounded with time. In this work, we develop a novel MBRL method (i) which relaxes the assumptions on the target transition model to belong to a generic family of mixture models; (ii) is applicable to large-scale training by incorporating a compression step such that the posterior estimate consists of a Bayesian coreset of only statistically significant past state-action pairs; and (iii) exhibits a sublinear Bayesian regret. To achieve these results, we adopt an approach based upon Stein's method, which, under a smoothness condition on the constructed posterior and target, allows distributional distance to be evaluated in closed form as the kernelized Stein discrepancy (KSD). The aforementioned compression step is then computed in terms of greedily retaining only those samples which are more than a certain KSD away from the previous model estimate. Experimentally, we observe that this approach is competitive with several state-of-the-art RL methodologies, and can achieve up-to 50 percent reduction in wall clock time in some continuous control environments.

show abstract

Online, Informative MCMC Thinning with Kernelized Stein Discrepancy

Cited by 2 publications

References 22 publications

Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning

Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning

Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning

Contact Info

Product

Resources

About