Importance Sampling and Necessary Sample Size: An Information Theory Approach

Abstract

Importance sampling approximates expectations with respect to a target measure by using samples from a proposal measure. The performance of the method over large classes of test functions depends heavily on the closeness between both measures. We derive a general bound that needs to hold for importance sampling to be successful, and relates the f -divergence between the target and the proposal to the sample size. The bound is deduced from a new and simple information theory paradigm for the study of importance sampling. As examples of the general theory we give necessary conditions on the sample size in terms of the Kullback-Leibler and χ 2 divergences, and the total variation and Hellinger distances. Our approach is non-asymptotic, and its generality allows to tell apart the relative merits of these metrics. Unsurprisingly, the nonsymmetric divergences give sharper bounds than total variation or Hellinger. Our results extend existing necessary conditions -and complement sufficient ones-on the sample size required for importance sampling.

Bayesian Update with Importance Sampling: Required Sample Size

“…The idea of Proposition 2 is inspired by [ 7 ], but adapted here from relative entropy to -divergence. Our results complement sufficient conditions on the sample size derived in [ 1 ] and necessary conditions for un-normalized (as opposed to autonormalized) importance sampling in [ 8 ]. In Section 3 , Proposition 4 gives a closed formula for the -divergence between posterior and prior in a linear-Gaussian Bayesian inverse problem setting.…”

“…The idea of Proposition 2 is inspired by [ 7 ], but adapted here from relative entropy to -divergence. Our results complement sufficient conditions on the sample size derived in [ 1 ] and necessary conditions for un-normalized (as opposed to autonormalized) importance sampling in [ 8 ].…”

“…Second, importance sampling in inverse problems, data assimilation and machine learning applications is often used as a building block of more sophisticated computational methods, and in such a case there may be little or no freedom in the choice of proposal. For this reason, throughout this paper we view both target and proposal as given and we focus on investigating the required sample size for accurate importance sampling with bounded test functions, following a similar perspective as [ 1 , 7 , 8 ]. The complementary question of how to choose the proposal to achieve a small variance for a given test function is not considered here.…”

“…Finally, in the Bayesian context the normalizing constant represents the marginal likelihood and is often computationally intractable. This motivates our focus on the autonormalized importance sampling estimator in ( 2 ), which estimates both and using Monte Carlo integration, as opposed to un-normalized variants of importance sampling [ 8 ].…”

Importance Sampling: Intrinsic Dimension and Computational Cost

“…The proof follows the approach of [37] for evaluating moments of ratios. Despite the complicated dependence of error constants on the problem at hand, there is further evidence for the centrality of the second moment ρ in the paper [91]. There it is shown (see Remark 4) that, when ρ is finite, a necessary condition for accuracy within the class of functions with bounded second moment under the proposal, is that the sample size N is of the order of the χ 2 divergence, and hence of the order of ρ.…”