Expanding the validity of the ensemble Kalman filter without the intrinsic need for inflation

Abstract

Abstract. The ensemble Kalman filter (EnKF) is a powerful data assimilation method meant for high-dimensional nonlinear systems. But its implementation requires somewhat ad hoc procedures such as localization and inflation. The recently developed finite-size ensemble Kalman filter (EnKF-N) does not require multiplicative inflation meant to counteract sampling errors. Aside from the practical interest in avoiding the tuning of inflation in perfect model data assimilation experiments, it also offers theoretical insights and a unique perspective on the EnKF. Here, we revisit, clarify and correct several key points of the EnKF-N derivation. This simplifies the use of the method, and expands its validity. The EnKF is shown to not only rely on the observations and the forecast ensemble, but also on an implicit prior assumption, termed hyperprior, that fills in the gap of missing information. In the EnKF-N framework, this assumption is made explicit through a Bayesian hierarchy. This hyperprior has so far been chosen to be the uninformative Jeffreys prior. Here, this choice is revisited to improve the performance of the EnKF-N in the regime where the analysis is strongly dominated by the prior. Moreover, it is shown that the EnKF-N can be extended with a normal-inverse Wishart informative hyperprior that introduces additional information on error statistics. This can be identified as a hybrid EnKF-3D-Var counterpart to the EnKF-N.

Chaotic dynamics and the role of covariance inflation for reduced rank Kalman filters with model error

“…The dynamical upwelling of model error differs from the sampling errors induced by nonlinear dynamics in perfect models, 25 treated in the modified EKF-AUS-NL (Palatella and Trevisan, 2015) and in the finite size ensemble Kalman filter, (EnKF-N) (Bocquet, 2011;Bocquet et al, 2015). Rather, we have shown that the upwelling of the unfiltered error through the Lyapunov filtration acts as a linear effect and is acute in the presence of additive model errors which are excited by transient instabilities.…”

“…For example, the linear form of EKF-AUS does not include the upwelling of unfiltered error in its estimated covariance -inclusion of multiplicative inflation to the estimated error covariance compensates for the upwelling of unfiltered errors which is not represented in the recursion for EKF-AUS, and simulates the terms (34b), (34c) and (34d) in the KF-AUSE recursion. Multiplicative inflation may also be used to account for mis-estimation of forecast errors resultant from nonlinear evolution, but this mis-estimation may also be accounted for 20 using less ad hoc approaches including parameterizing this error with hyperpriors (Bocquet et al, 2015). We argue that the hyperprior in EnKF-N can, in principle, also be selected to take into account the structure of the ideal posterior for the reduced rank estimator, in the presence of model error, described by KF-AUSE (see also the discussion at the end of section 4.2).…”

“…This hypothesis is supported by our results and the above discussion: the combination of rank deficiency of the analysis and the presence of additive model error determines an intrinsic role for covariance inflation in ensemble based Kalman filters in chaotic, dynamical systems, due to the upwelling of unfiltered errors. However, our above discussion also highlights how the need for inflation can be mitigated by: (i) sufficiently increasing the ensemble size ; (ii) rectifying the rank deficiency of the analysis update via hybridization (Penny, 2017); (iii) utilizing a hyperprior 30 which takes into account the dynamical upwelling and mis-estimation of error (Bocquet et al, 2015); or (iv) some combination of the above. In the following section, we numerically explore the effects of stability, transient instability, and the strength therein, on filter divergence and the need for covariance inflation with reduced rank estimators.…”

“…Particularly, we may understand the covariance of KF-AUSE to express the ideal forecast error of the rank deficient EnKF, and should guide any independent, identically distributed (iid) sampling scheme which represents this error 10 distribution. The hyperprior in EnKF-N (Bocquet et al, 2015), used to describe the mis-estimation of the true error statistics from a finite iid draw of samples, may also incorporate the structure of the ideal posterior under a rank deficient gain. Similarly to how the hyperprior for the covariance in EnKF-N is restricted to the cone of positive semi-definite symmetric matrices in perfect models, its extension to additive model error may incorporate a restriction to the covariance matrices which share the stratification of uncertainty expressed in KF-AUSE.…”

Quasi static ensemble variational data assimilation

“…, 1, 20). Sampling errors are systematically accounted for using the IEnKS finite-size version 5 (Bocquet, 2011;Bocquet and Sakov, 2012;Bocquet et al, 2015) which avoids the need for inflation and its costly tuning.…”