Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems

Abstract

It has been widely realized that Monte Carlo methods (approximation via a sample ensemble) may fail in large scale systems. This work offers some theoretical insight into this phenomenon in the context of the particle filter. We demonstrate that the maximum of the weights associated with the sample ensemble converges to one as both the sample size and the system dimension tends to infinity. Specifically, under fairly weak assumptions, if the ensemble size grows sub-exponentially in the cube root of the system dimension, the convergence holds for a single update step in state-space models with independent and identically distributed kernels. Further, in an important special case, more refined arguments show (and our simulations suggest) that the convergence to unity occurs unless the ensemble grows super-exponentially in the system dimension.The weight singularity is also established in models with more general multivariate likelihoods, e.g. Gaussian and Cauchy. Although presented in the context of atmospheric data assimilation for numerical weather prediction, our results are generally valid for high-dimensional particle filters.

Exploring Practical Estimates of the Ensemble Size Necessary for Particle Filters

“…In a series of related papers, Bengtsson et al (2008), Bickel et al (2008), and Snyder et al (2008) show that the curse of dimensionality is also manifest in the simplest particle filter. They demonstrate that the required ensemble size scales exponentially with a statistic related, in part, to the system dimension and that may be considered as an effective dimension.…”

“…In the case examined by Bengtsson et al (2008), Bickel et al (2008), and Snyder et al (2008), the proposal is the transition distribution for the system dynamics, where new particles are generated by evolving particles from the previous time under the system dynamics. It yields the bootstrap filter of Gordon et al (1993) and was termed the ''standard'' proposal by Snyder (2012) and Snyder et al (2015).…”

“…Morzfeld et al 2012). An alternative is to utilize the results of Bengtsson et al (2008), Bickel et al (2008), and Snyder et al (2008Snyder et al ( , 2015, which relate the behavior of the weights in the linear, Gaussian case to eigenvalues of certain covariance matrices that may be estimated from an existing ensemble. We aim to evaluate the use of these results in a more general nonlinear, non-Gaussian setting, by using ensembles from a working sequential EnKF to calculate the relevant statistics (without implementing the particle filter directly.)…”

An efficient multiple particle filter based on the variational Bayesian approach

“…A number of judicious choices of importance densities, along with various resampling strategies have been proposed in an attempt to tackle this issue [2], [6]. Although many of these have provided satisfactory results in a number of low-dimensional state-space applications [4], [6], they nevertheless remain inefficient in the cases of very large dimensional system [7]- [11]. This is mainly due to the fact that the number of particles required to sufficiently sample the state-space needs to be very large; in some special situations it has been shown that the required number of particles scales exponentially with the system dimension (see e.g.…”

Scaling up data mining algorithms: review and taxonomy

“…Most data mining algorithms suffer from the "curse of dimensionality" [9], which means that their performance on any task deteriorates, sometimes very quickly, as the dimensionality of the problem increases [127]. Thus, scaling up a data mining algorithm not only involves improving its speed but also maintaining the quality of its solution.…”