State estimation and prediction using clustered particle filters

Abstract

Particle filtering is an essential tool to improve uncertain model predictions by incorporating noisy observational data from complex systems including non-Gaussian features. A class of particle filters, clustered particle filters, is introduced for high-dimensional nonlinear systems, which uses relatively few particles compared with the standard particle filter. The clustered particle filter captures non-Gaussian features of the true signal, which are typical in complex nonlinear dynamical systems such as geophysical systems. The method is also robust in the difficult regime of high-quality sparse and infrequent observations. The key features of the clustered particle filtering are coarse-grained localization through the clustering of the state variables and particle adjustment to stabilize the method; each observation affects only neighbor state variables through clustering and particles are adjusted to prevent particle collapse due to high-quality observations. The clustered particle filter is tested for the 40-dimensional Lorenz 96 model with several dynamical regimes including strongly nonGaussian statistics. The clustered particle filter shows robust skill in both achieving accurate filter results and capturing nonGaussian statistics of the true signal. It is further extended to multiscale data assimilation, which provides the large-scale estimation by combining a cheap reduced-order forecast model and mixed observations of the large-and small-scale variables. This approach enables the use of a larger number of particles due to the computational savings in the forecast model. The multiscale clustered particle filter is tested for one-dimensional dispersive wave turbulence using a forecast model with model errors.data assimilation | particle filter | non-Gaussian | uncertainty quantification D ata assimilation or filtering combines numerical forecast models with observational data to provide the best statistical estimation and prediction of complex systems. Due to the high dimensionality of complex nonlinear systems such as geophysical systems, accurate and efficient estimation and prediction of such complex systems are formidable tasks. They require enormously large computational resources to run forecast models and observations can be sparse and infrequent, as in oceanography. As a Monte Carlo approach, ensemble-based methods (1, 2) combined with covariance inflation and localization are indispensable tools because they allow computationally cheap, low-dimensional ensemble state approximations for the systems. They have performed well for operational applications such as numerical weather prediction (3, 4). Nevertheless, because the ensemble-based methods approximate the forecast distribution using Gaussian statistics, when the ensemble is not sufficiently large these methods can lead to inaccurate estimation and prediction when the true signal has non-Gaussian statistics, which are typical for a wide range of systems. Particle filtering captures non-Gaussian features using different weights for different...

Variational particle smoothers and their localization

“…Small observation-error covariances have been one shortcoming of the local PF (with standard proposal) in the past, e.g. Lee and Majda (2016), and our computations with linear models suggest that they can be overcome by using optimal rather than standard proposals.…”

“…This is the main idea behind localization of PFs, first discussed by Bengtsson et al () and van Leeuwen (), which typically consists of the following two steps (e.g. Lei and Bickel, ; Reich, ; Penny and Miyoshi, ; Poterjoy, ; Tödter and Ahrens, ; Lee and Majda, ; Poterjoy and Anderson, ; Poterjoy et al give specific localization strategies): find a way to compute weights in Equation locally; make use of these local weights without upsetting the complex multivariate relationships between variables (model “balance”). …”

“…This is the main idea behind localization of PFs, first discussed by Bengtsson et al (2003) and van Leeuwen (2003), which typically consists of the following two steps (e.g. Lei and Bickel, 2011;Reich, 2013;Penny and Miyoshi, 2015;Poterjoy, 2015;Tödter and Ahrens, 2015;Lee and Majda, 2016;Poterjoy and Anderson, 2016;Poterjoy et al, 2017 give specific localization strategies):…”

“…The main idea of "localizing" PFs is to exploit banded problem structure to avoid PF collapse (section 2.4 below). Several methods to localize PFs have been invented and have been shown to "work well", mostly on relatively simple models (Lei and Bickel, 2011;Reich, 2013;Penny and Miyoshi, 2015;Poterjoy, 2015;Tödter and Ahrens, 2015;Lee and Majda, 2016;Poterjoy et al, 2017), but Poterjoy and Anderson (2016) and Robert et al (2017) present results in a realistic NWP context.…”

Localization for MCMC: sampling high-dimensional posterior distributions with local structure

“…We already brought up NWP and the EnKF as an example of a local problem in which localization enables efficient computations in high dimensional problems. Similarly, exploiting localization in importance sampling (particle filtering) is also a current topic in NWP, see, e.g., [34,35,44,[46][47][48][49][50]60,62]. NWP, however, is usually not considered an inverse problem due to its sequential-in-time nature.…”