Non‐Gaussian Detection Using Machine Learning With Data Assimilation Applications

Abstract

The assumption that variables, and their errors, are Gaussian distributed is commonplace in areas such as numerical weather prediction and modeling. Research such as that undertaken by Perron and Sura (2013) has shown that this assumption is generally false for atmospheric variables, and that Gaussian variables in the atmosphere are rare. The aforementioned statement was based on a 62 year long project from daily data taken from the National Centers for Environmental Prediction and the National Center for Atmospheric Research, using the Reanalysis I Project data set. Given this evidence, the need to be able to relax the Gaussian assumption for the errors involved in the data assimilation schemes becomes quite important if the analysis error is to be minimized. By doing so, we may deliver an improvement in the subsequent forecast.Most of the current formulations of data assimilation, for example, variational methods such as 3DVar and 4DVar (which are based upon Bayes theorem Fletcher, 2017), and ensemble methods such as the Ensemble Kalman Filter (EnKF;Evensen & Van Leeuwen, 1996), the (local) Ensemble Transform Kalman Filter ((L)ETKF) Ott et al. (2004); Wang and Bishop (2003), which are based upon a control theory/weighted least squares approach using ensemble members to approximate the analysis mean and covariances, and the Maximum Likelihood Ensemble Kalman Filter, Zupanski (2005), which uses the Kalman filter equations combined with the 3DVar cost function, all assume that the errors involved are Gaussian distributed. Also in linear state estimation, the initial condition x 0 is also assumed to be Gaussian distributed. Other papers that look into non-Gaussian data assimilation methods are local particle filters, van Leeuwen et al. (2019), andLeeuwen (2014) who looked into Gaussian anamorphosis on the EnKF. However, at the Cooperative Institute for Research in the Atmosphere (CIRA) at Colorado State University, a theory has been developed and tested, that allows for the Gaussian assumption for the distribution of the errors to be relaxed to a lognormal distribution. In Fletcher and Zupanski (2006a), the theory is presented for the case

A dynamical Gaussian, lognormal, and reverse lognormal Kalman filter

“…To test the implementation of the dynamical mixed Kalman filter, we perform some experiments with a toy model. We choose the Lorenz‐63 model (Lorenz, 1963), which has been shown to contain lognormal and reverse lognormal signals (Goodliff et al ., 2020; 2022; 2023). The governing differential equations are given by $$$$ \left\{\begin{array}{ccc}& \frac{dx}{dt}=\sigma \left(y-x\right),\hfill & \hfill \\ {}& \frac{dy}{dt}=\rho x-y- xz,\hfill & \\ {}& \frac{dz}{dt}= xy-\beta z,\hfill & \end{array}\right.$$…”

“…Some sort of decision function is necessary to determine which components need to be transformed. How to define this decision function is still an open question for most problems, although advances have been made with machine-learning techniques (Goodliff et al, 2020(Goodliff et al, , 2022(Goodliff et al, , 2023.…”

“…In order to assess the dynamical features of our mixed Kalman filter, a decision function is necessary to select the correct distribution to use for the z component at every assimilation step. We follow a technique similar to the one developed by Goodliff et al (2022), based on machine-learning methods. First, a long control run for t ∈ [0, 1000] is generated from the initial conditions x 0 = (−3, −3, 20) with time step dt = 0.01.…”

“…If the z-score is larger than 1, we assume the errors of the z variable of the Lorenz model to be lognormally distributed; when it is smaller than −1, we assume a reverse lognormal distribution; if the null hypothesis is accepted, Gaussian errors are assumed. The significance level is chosen in order to produce enough cases where lognormal and reverse lognormal errors are predicted, and is reproduced from Goodliff et al (2022). For example, for 𝛼 = 0.3, the z component behaves lognormally about 40% of the time and reverse lognormally about 15% of the time.…”

“…In the case of lognormal errors, variational data assimilation methods are already well established (Fletcher & Zupanski, 2006a, 2006bFletcher, 2010Fletcher, , 2022 and machine learning has been suggested as a promising way of selecting the underlying error distribution (Goodliff et al, 2020;Goodliff et al, 2022). Recently, Fletcher et al (2023) showed that it is also possible to adapt the Kalman filter to allow for lognormal errors.…”