Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature

Abstract

Abstract. Both the root mean square error (RMSE) and the mean absolute error (MAE) are regularly employed in model evaluation studies. Willmott and Matsuura (2005) have suggested that the RMSE is not a good indicator of average model performance and might be a misleading indicator of average error, and thus the MAE would be a better metric for that purpose. While some concerns over using RMSE raised by Willmott and Matsuura (2005) and Willmott et al. (2009) are valid, the proposed avoidance of RMSE in favor of MAE is not the solution. Citing the aforementioned papers, many researchers chose MAE over RMSE to present their model evaluation statistics when presenting or adding the RMSE measures could be more beneficial. In this technical note, we demonstrate that the RMSE is not ambiguous in its meaning, contrary to what was claimed by Willmott et al. (2009). The RMSE is more appropriate to represent model performance than the MAE when the error distribution is expected to be Gaussian. In addition, we show that the RMSE satisfies the triangle inequality requirement for a distance metric, whereas Willmott et al. (2009) indicated that the sums-ofsquares-based statistics do not satisfy this rule. In the end, we discussed some circumstances where using the RMSE will be more beneficial. However, we do not contend that the RMSE is superior over the MAE. Instead, a combination of metrics, including but certainly not limited to RMSEs and MAEs, are often required to assess model performance.

A fully coupled atmosphere–ocean wave modeling system for the Mediterranean Sea: interactions and sensitivity to the resolved scales and mechanisms

“…Although WEW increases the wind and the SWH underestimation, it overall improves the SWH RMSE by approximately 7 % against buoys data and by 11 % against remotely sensed data. In contrast to the bias scores, RMSE penalizes the variance between in situ or remotely sensed data and the simulations implying a deterioration of the RMSE in CTRL run (Chai and Draxler, 2014). Similar RMSE improvements by the coupled systems have been also confirmed in the relevant literature (e.g., Lionello et al, 2003;Renault et al, 2012).…”

Measurements of wave damping by a grease ice slick in Svalbard using off-the-shelf sensors and open-source electronics

“…R 2 is a number between 0 and 1 that indicates the part of the variance in the data that are explained by the model, so that an R 2 value of 1 indicates that the model fully describes the variance in the data, while an R 2 value of 0 indicates that the model fails in explaining anything of the data presented. The MAE and RMSE are other well-established metrics that can be used to quantify the quality of a model (Willmott, 1982;Chai and Draxler, 2014), and that directly measure the discrepancy between model predictions and experimental data.…”

On the kernel and particle consistency in smoothed particle hydrodynamics

“…(32) are also used to assess the errors in the estimates of the derivatives. Since the SPH errors are expected to have a normal rather than a uniform distribution, the RMSE will provide a better representation of the error distribution than other statistical metrics [3]. Compared to the mean absolute error (MAE), which is more closely related to the L 1 -norm, the RMSE gives a higher weighting toward large errors in the sample than the mean average error and therefore it is superior at revealing model performance differences.…”