Penalized ensemble Kalman filters for high dimensional non-linear systems

Abstract

The ensemble Kalman filter (EnKF) is a data assimilation technique that uses an ensemble of models, updated with data, to track the time evolution of a usually non-linear system. It does so by using an empirical approximation to the well-known Kalman filter. However, its performance can suffer when the ensemble size is smaller than the state space, as is often necessary for computationally burdensome models. This scenario means that the empirical estimate of the state covariance is not full rank and possibly quite noisy. To solve this problem in this high dimensional regime, we propose a computationally fast and easy to implement algorithm called the penalized ensemble Kalman filter (PEnKF). Under certain conditions, it can be theoretically proven that the PEnKF will be accurate (the estimation error will converge to zero) despite having fewer ensemble members than state dimensions. Further, as contrasted to localization methods, the proposed approach learns the covariance structure associated with the dynamical system. These theoretical results are supported with simulations of several non-linear and high dimensional systems.

Interpretable and Scalable Graphical Models for Complex Spatio-temporal Processes

“…Most similar to our proposed work is Hou et al (2021), which suggests to imple-ment EnKF with a sparse inverse covariance estimator to handle the high-dimensional regime. However, we note that many real-world processes are complex and generating heterogeneous multiway/tensor-variate data.…”

“…Here we propose incorporating the multiway covariance / inverse covariance models into the penalized EnKF framework of Hou et al (2021).…”

“…Due to the non-homogeneous nature of these data, estimation of the second-order information that encodes (conditional) dependency structure within the data is of great importance. Assuming the data are drawn from a tensor normal distribution, a straightforward way to estimate this structure is to vectorize the tensor and estimate the underlying Gaussian graphical model associated with the vector, as suggested by Hou et al (2021). Such an approach ignores the tensor structure and requires estimating a rather high dimensional precision matrix, often with insufficient sample size.…”

TensorGraphicalModels: A Julia toolbox for multiway covariance models and ensemble Kalman filter

“…It does so by using an empirical approximation to the well-known Kalman filter. Specifically, we implement a penalized version (PEnKF) proposed by [15,34] that use sparse estimators for the state (inverse) covariance, which constitutes an important step of the 1 Note that Tlasso, TeraLasso, Syglasso/SG-PALM are generalizable to precision matrices of the form filtering algorithm. The sparse and Kronecker-structure regularization allow for both an efficient computation and hence fast temporal evolution of the system and elimination of spurious correlations [35] that are often present in high-dimensional regime.…”

Assimilation of Observations from Meteorological Satellites in the Hydrometcenter of Russia

“…(Ueno and Tsuchiya, 2009) proposed to regularize the sample covariance matrix by imposing a sparse structure in the inverse covariance (precision) matrix. A similar approach is taken by Hou et al (2021). Gryvill and Tjelmeland (2023) imposed a Markov random field structure to obtain a sparse prior precision matrix.…”