Dynamic Time Warping as a New Evaluation for Dst Forecast With Machine Learning

Abstract

Models based on neural networks and machine learning are seeing a rise in popularity in space physics. In particular, the forecasting of geomagnetic indices with neural network models is becoming a popular field of study. These models are evaluated with metrics such as the root-mean-square error (RMSE) and Pearson correlation coefficient. However, these classical metrics sometimes fail to capture crucial behavior. To show where the classical metrics are lacking, we trained a neural network, using a long short-term memory network, to make a forecast of the disturbance storm time index at origin time t with a forecasting horizon of 1 up to 6 h, trained on OMNIWeb data. Inspection of the model's results with the correlation coefficient and RMSE indicated a performance comparable to the latest publications. However, visual inspection showed that the predictions made by the neural network were behaving similarly to the persistence model. In this work, a new method is proposed to measure whether two time series are shifted in time with respect to each other, such as the persistence model output versus the observation. The new measure, based on Dynamical Time Warping, is capable of identifying results made by the persistence model and shows promising results in confirming the visual observations of the neural network's output. Finally, different methodologies for training the neural network are explored in order to remove the persistence behavior from the results.

An operational approach to forecast the Earth’s radiation belts dynamics

“…Based on this algorithm, Frías-Paredes et al ( 2016) propose the Temporal Distortion Index (TDI), which indicates to what extent the two-time series are systematically (or not) late (or early). Unlike the approach proposed by Laperre et al (2020), the TDI does not indicate the value of a possible systematic time lag, but whether the two-time series exhibit this type of behaviour and to which extent. In return, there is no need for several computations of the DTW measure as only one (per forecast horizon) is sufficient to get the TDI.…”

“…Let us note that LSTMs have already demonstrated a good efficiency on geomagnetic index prediction problems (see, e.g., Gruet et al, 2018;Chakraborty & Morley, 2020;Laperre et al, 2020). Since this is the first study that focuses on the forecast of the Ca index, there is no immediate baseline for us to compare our model to.…”

“…Some studies, such as Wintoft & Wik (2018) and Laperre et al (2020), highlight the fact that some forecasting models, that display a great RMSE or Pearson correlation actually fail to reliably forecast high disturbance periods in advance. Laperre et al (2020) show that some prediction models exhibit systematic time lags between the observed time series and the predicted time series. This systematic time lag would most often be of the order of magnitude of the model's prediction horizon.…”

The Dynamic Time Warping as a means to assess modeled solar wind time series

“…The sequences are eventually non-linearly warped along the time dimension to match each other (Müller 2007;Górecki & Luczak 2013). A recent study by Laperre et al (2020) used this technique for evaluating the Dst forecast with machine learning. In this work, we will apply the DTW technique for the first time as a means to quantify differences between observed and modeled time series in solar wind forecasting.…”

Domain of Influence Analysis: Implications for Data Assimilation in Space Weather Forecasting

“…However, empirical models lack the sophistication of more expensive first-principles based numerical models. Recently, machine learning methods have emerged, that can provide a new approach to space weather forecasting (Camporeale, 2019;Laperre et al, 2020). Most of these methods, while promising, must still undergo extensive validation.…”