A Lagged Particle Filter for Stable Filtering of Certain High-Dimensional State-Space Models

Ruzayqat, Hamza; Er-Raiy, Aimad; Beskos, Alexandros; Crisan, Dan; Jasra, Ajay; Kantas, Nikolas

doi:10.1137/21m1450392

Cited by 10 publications

(6 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As discussed in Johansen (2015) and Guarniero et al (2017) the smoothing and filtering distributions (i.e. p(x 1:t |y 1:t ) and p(x t |y 1:t ), respectively) have significantly different support in the presence of informative observations, especially in high di-mensional settings and so we expect that including the influence of future observations in the proposals and targets in Section 3.2 (as in lookahead methods, e.g., Lin et al (2013); Guarniero et al (2017); Ruzayqat et al (2022)) would lead to considerable improvements in the accuracy of the estimates and might ultimately be essential in the development of good general purpose filters for high dimensional problems. This work focuses on obtaining approximations of the filtering distribution for high dimensional SSM.…”

Section: Discussionmentioning

confidence: 99%

A Divide and Conquer Sequential Monte Carlo Approach to High Dimensional Filtering

Crucinio¹,

Johansen²

2023

STAT SINICA

View full text Add to dashboard Cite

We propose a divide-and-conquer approach to filtering which decomposes the state variable into low-dimensional components to which standard particle filtering tools can be successfully applied and recursively merges them to recover the full filtering distribution. It is less dependent upon factorization of transition densities and observation likelihoods than competing approaches and can be applied to a broader class of models. Performance is compared with state-of-the-art methods on a benchmark problem and it is demonstrated that the proposed method is broadly comparable in settings in which those methods are applicable, and that it can be applied in settings in which they cannot.

show abstract

Section: Discussionmentioning

confidence: 99%

A Divide and Conquer Sequential Monte Carlo Approach to High Dimensional Filtering

Crucinio¹,

Johansen²

2023

STAT SINICA

View full text Add to dashboard Cite

show abstract

“…An alternative course of action is to aim for methods that aim to approximate 𝜋 k without using importance sampling and resampling for Π k and Equation (2). The method in Ruzayqat et al (2022), which was designed for high-dimensional filtering, has been our first attempt to perform filtering for this model. The method transports particles from a variant of Equation ( 2) that only considers the path between t k−L+1 to t k for a small lag L and thus bypasses the degeneracy issues by introducing a moderately low bias.…”

Section: 22mentioning

confidence: 99%

“…Unfortunately, simple or standard designs of PFs require an exponential cost in the dimension d to achieve a certain precision and hence are impractical for very high-dimensional applications. Several methods Beskos et al (2014aBeskos et al ( , 2014bBeskos et al ( , 2017; Kantas et al (2014);Pons Llopis et al (2018); Ruzayqat et al (2022) based on a combination of sequential Monte Carlo (SMC: e.g., Del Moral et al (2006);Del Moral (2013)) and MCMC can be adopted. For well-designated classes of models, they have been shown to be both mathematically and empirically able to deal with high-dimensional filtering, at a cost that is polynomial in the dimension of the problem.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sequential Markov chain Monte Carlo for Lagrangian data assimilation with applications to unknown data locations

Ruzayqat,

Beskos,

Crisan

et al. 2024

Quart J Royal Meteoro Soc

Self Cite

View full text Add to dashboard Cite

We consider a class of high‐dimensional spatial filtering problems, where the spatial locations of observations are unknown and driven by the partially observed hidden signal. This problem is exceptionally challenging, as not only is it high‐dimensional, but the model for the signal yields longer‐range time dependences through the observation locations. Motivated by this model, we revisit a lesser‐known and provably convergent computational methodology from Berzuini et al. (1997, Journal of the American Statistical Association, 92, 1403–1412); Centanniand Minozzo (2006, Journal of the American Statistical Association, 101, 1582–1597); Martin et al. (2013, Annals of the Institute of Statistical Mathematics, 65, 413–437) that uses sequential Markov Chain Monte Carlo (MCMC) chains. We extend this methodology for data filtering problems with unknown observation locations. We benchmark our algorithms on linear Gaussian state‐space models against competing ensemble methods and demonstrate a significant improvement in both execution speed and accuracy. Finally, we implement a realistic case study on a high‐dimensional rotating shallow‐water model (of about – dimensions) with real and synthetic data. The data are provided by the National Oceanic and Atmospheric Administration (NOAA) and contain observations from ocean drifters in a domain of the Atlantic Ocean restricted to the longitude and latitude intervals , , respectively.

show abstract

“…Traditionally the main difficulty with particle filtering techniques has been the 'curse of dimensionality' (Bengtsson et al, 2008;Bickel et al, 2008;Snyder, 2011), where in high dimensional settings filtering leads to degeneracy of the importance weights associated with each particle and loss of diversity within an ensemble. To improve the applicability of PFs there have been many recent developments involving; localisation techniques (e.g., the reviews in Farchi and Bocquet (2018); Graham and Thiery (2019)), incorporation of tempering/mutation steps (e.g., Cotter et al (2020); Ruzayqat et al (2022)), hybrid approaches, improved computational implementations and combination of the above with improved proposal distributions. The above ongoing efforts have extended the applicability of PF methods within geoscientific domains.…”

mentioning

confidence: 99%

ParticleDA.jl v.1.0: A real-time data assimilation software platform

Giles

Graham

Giordano

et al. 2023

Preprint

View full text Add to dashboard Cite

Abstract. Digital twins of physical and human systems informed by real-time data, are becoming ubiquitous across weather forecasting, disaster preparedness, and urban planning, but researchers lack the tools to run these models effectively and efficiently, limiting progress. One of the current challenges is to assimilate observations in highly nonlinear dynamical systems, as the practical need is often to detect abrupt changes. We developed a software platform to improve the use of real-time data in highly nonlinear system representations where non-Gaussianity prevents the use of more standard Data Assimilation. Optimal Particle filtering data assimilation (DA) techniques have been implemented within an user-friendly open source software platform in Julia – ParticleDA.jl. To ensure the applicability of the developed platform in realistic scenarios, emphasis has been placed on numerical efficiency, scalability and optimisation for high performance computing frameworks. Furthermore, the platform has been developed to be forward model agnostic, ensuring that it is applicable to a wide range of modelling settings, for instance unstructured and non-uniform meshes in the spatial domain or even state spaces that are not spatially organised. Applications to tsunami and numerical weather prediction demonstrate the computational benefits in terms of lower errors, lower computational costs (due to ensemble size and the algorithm's overheads being minimised) and versatility thanks to flexible I/O in a high level language Julia.

show abstract

A Lagged Particle Filter for Stable Filtering of Certain High-Dimensional State-Space Models

Cited by 10 publications

References 25 publications

A Divide and Conquer Sequential Monte Carlo Approach to High Dimensional Filtering

A Divide and Conquer Sequential Monte Carlo Approach to High Dimensional Filtering

Sequential Markov chain Monte Carlo for Lagrangian data assimilation with applications to unknown data locations

ParticleDA.jl v.1.0: A real-time data assimilation software platform

Contact Info

Product

Resources

About