Mean Squared Error, Deconstructed

Abstract

As science becomes increasingly cross‐disciplinary and scientific models become increasingly cross‐coupled, standardized practices of model evaluation are more important than ever. For normally distributed data, mean squared error (MSE) is ideal as an objective measure of model performance, but it gives little insight into what aspects of model performance are “good” or “bad.” This apparent weakness has led to a myriad of specialized error metrics, which are sometimes aggregated to form a composite score. Such scores are inherently subjective, however, and while their components may be interpretable, the composite itself is not. We contend that, a better approach to model benchmarking and interpretation is to decompose MSE into interpretable components. To demonstrate the versatility of this approach, we outline some fundamental types of decomposition and apply them to predictions at 1,021 streamgages across the conterminous United States from three streamflow models. Through this demonstration, we hope to show that each component in a decomposition represents a distinct concept, like “season” or “variability,” and that simple decompositions can be combined to represent more complex concepts, like “seasonal variability,” creating an expressive language through which to interrogate models and data.

Mono Camera-based Human Skeletal Tracking for Squat Exercise Abnormality Detection using Double Exponential Smoothing

“…This showed that the subject was able to maintain a better squat form while performing the exercise in normal conditions compared to an abnormal condition. This proves that MSE is a suitable measurement for evaluating the performance of the system [27]. The smoothed value also helps in evaluating the data by reducing the noise.…”

Forecast calibrations of surface air temperature over Xinjiang based on U-net neural network

“…All these error metrics aggregate the error evaluation results into composite scores, which lack interpretability. To solve this problem, based on the error decomposition method proposed by Hodson et al (2021), the mean square error is decomposed into three interpretable error components in this study, namely, a bias term, a distribution term and a sequence term. Error diagnosis of each model is then carried out to indicate the direction required for further optimization.…”

“…All the evaluations discussed in Section 3.1 aggregate the calibrations into composite metrics, without understanding which aspects of the model performance are "good" or "bad". In this section, based on the error decomposition method proposed by Hodson et al (2021), the MSE is decomposed into three components, each representing a distinct concept. The bias term quantifies how well the model reproduces the mean of the observation.…”

“…Previous studies mostly aggregate the results of error evaluation to form composite scores, such as the Error decomposition is important in the interpretation and composition of error metrics, playing a prominent role in the earth sciences (Murphy, 1988;Gupta et al, 2009). In this study, based on the decomposition method proposed by Hodson et al (2021), MSE is decomposed into three interpretable components to comprehensively evaluate the models, diagnose the sources of errors in the models and reveal the correction effects on errors from different sources to determine which aspects of the models require further revision. This paper uses the U-net framework to correct the biases in forecasts of the surface air temperature (surface air temperature) from the Global Ensemble Forecasting System (GEFS) of the NCEP/ NOAA.…”

“…The relationship between each evaluation metric and forecast skill. method proposed byHodson et al (2021). The derivation begins by monotonically sorting the model forecasts and observations, then the new error is calculated:…”