Forecast evaluation for data scientists: common pitfalls and best practices

Abstract

Recent trends in the Machine Learning (ML) and in particular Deep Learning (DL) domains have demonstrated that with the availability of massive amounts of time series, ML and DL techniques are competitive in time series forecasting. Nevertheless, the different forms of non-stationarities associated with time series challenge the capabilities of data-driven ML models. Furthermore, due to the domain of forecasting being fostered mainly by statisticians and econometricians over the years, the concepts related to forecast evaluation are not the mainstream knowledge among ML researchers. We demonstrate in our work that as a consequence, ML researchers oftentimes adopt flawed evaluation practices which results in spurious conclusions suggesting methods that are not competitive in reality to be seemingly competitive. Therefore, in this work we provide a tutorial-like compilation of the details associated with forecast evaluation. This way, we intend to impart the information associated with forecast evaluation to fit the context of ML, as means of bridging the knowledge gap between traditional methods of forecasting and adopting current state-of-the-art ML techniques.We elaborate the details of the different problematic characteristics of time series such as non-normality and non-stationarities and how they are associated with common pitfalls in forecast evaluation. Best practices in forecast evaluation are outlined with respect to the different steps such as data partitioning, error calculation, statistical testing, and others. Further guidelines are also provided along selecting valid and suitable error measures depending on the specific characteristics of the dataset at hand.

Deriving Input Variables through Applied Machine Learning for Short-Term Electric Load Forecasting in Eskilstuna, Sweden

“…For every full-year run, 86,520 errors (multiply the forecast horizon by the total number of predictions) were analyzed, together with 515 average errors (one for each prediction made), and a single average error. Which error metric to be used for different datasets can be derived from Hewamalage et al [37]. The Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) were concluded to be the two most common ones used for the STLF of an electrical load in Nti et al [38], and the former was used in this study and presented in the MW.…”

“…The model was trained, and predictions were made according to the inputs, and at some points, it was re-trained. Historical Forecasts uses a rolling window approach for the rolling origin evaluation of the forecast [37]. Each prediction, error, and error metric are saved for further analysis.…”

Google Trends Forecasting: Model re-specification of time-series for enhancing forecasts of youth unemployment

“…With model recalibration, all model parameters remain constant, meaning that the forecasting model does not fundamentally change. However, unlike the updating of the forecasting model, recalibrating the forecasting model results in all parameters to be reestimated based on the updated window (Hewamalage et al, 2023;Petropoulos et al, 2022;Tashman, 2000). As a result, the forecasting model is ensured to have the best possible fit with the new data (Hewamalage et al, 2023;Tashman, 2000).…”

Modeling information diffusion in social media: data-driven observations

“…Certainly having just one metric is not acceptable for the purposes of our task, given all of the different properties or key regularities we would like our models to capture. The choice of right error metrics is critical, even more so in this domain, where problematic time series characteristics such as non-normalities or non-stationarities are often present in the data (Hewamalage et al, 2022 ). These characteristics can make some error measurements susceptible to break down, which can result in spurious conclusions about model performance.…”