Basic Statistical Estimation Outperforms Machine Learning in Monthly Prediction of Seasonal Climatic Parameters

Abstract

Machine learning (ML) has been utilized to predict climatic parameters, and many successes have been reported in the literature. In this paper, we scrutinize the effectiveness of five widely used ML algorithms in the monthly prediction of seasonal climatic parameters using monthly image data. Specifically, we quantify the predictive performance of these algorithms applied to five climatic parameters using various combinations of features. We compare the predictive accuracy of the resulting trained ML models to that of basic statistical estimators that are computed directly from the training data. Our results show that ML never significantly outperforms the statistical baseline, and underperforms for most feature sets. Unlike previous similar studies, we provide error bars for the relative performance of different predictors based on jackknife estimates applied to differences in predictive error magnitudes. We also show that the practice of shuffling data sequences which was employed in some previous references leads to data leakage, resulting in over-estimated performance. Ultimately, the paper demonstrates the importance of using well-grounded statistical techniques when producing and analyzing the results of ML predictive models.

Comparison of Phenolic Content and Antioxidant Activity for Fermented and Unfermented Rooibos Samples Extracted with Water and Methanol

“…Some references that apply ML to food chemistry do not compare their results to a statistical baseline, so it is impossible to tell whether ML actually worked better than statistics [ 27 , 32 , 59 , 60 , 61 ]. Furthermore, ML is well-known for being prone to misapplication and misinterpretation [ 36 , 37 , 38 ]. For example, the good results from ML reported in [ 4 , 31 ] are in fact due to data leakage: different measurements from the same sample were included in both training and testing sets.…”

“…Recently, several authors in the plant chemistry field have employed ML techniques to discover associations between chemical properties and food characteristics [ 4 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ]. At the same time, some authors have cast doubt on the efficacy of ML when relatively small samples are used [ 36 , 37 , 38 ] (e.g., less than a few hundred), which is typically the case in food chemistry applications. For this reason, in addition to conventional methods, we also apply ML to the problem of classifying samples into FR and UFR based on the phenolic conetent and antioxidant activities, and do a rigorous comparison of the accuracy of these methods compared to standard statistical techniques.…”

Feature Reduction for the Classification of Bruise Damage to Apple Fruit Using a Contactless FT-NIR Spectroscopy with Machine Learning

“…The precision of the different classifiers were evaluated using the testing set. In order to obtain error bars, jackknife with leave-out-one was implemented [32,33]. For the training set error, the left-out instances were 1/3 of the training set, while for the testing set all instances were left out one-by-one to obtain the jacknife estimate of the standard deviation.…”

Forecasting the June Ridge Line of the Western Pacific Subtropical High with a Machine Learning Method

“…In recent years, machine learning methods have been widely used for climate forecast systems and have obviously improved the forecast beyond the seasonal time scale [22][23][24][25][26][27][28][29][30]. However, these studies mostly focus on the prediction of sea surface temperature (SST) indexes (e.g., El Niño-Southern Oscillation, ENSO).…”