Probabilistic performance estimators for computational chemistry methods: Systematic improvement probability and ranking probability matrix. II. Applications

Abstract

In the first part of this study (Paper I), we introduced the systematic improvement probability (SIP) as a tool to assess the level of improvement on absolute errors to be expected when switching between two computational chemistry methods. We developed also two indicators based on robust statistics to address the uncertainty of ranking in computational chemistry benchmarks: P inv , the inversion probability between two values of a statistic, and P r , the ranking probability matrix. In this second part, these indicators are applied to nine data sets extracted from the recent benchmarking literature. We illustrate also how the correlation between the error sets might contain useful information on the benchmark dataset quality, notably when experimental data are used as reference.

Uncertainty quantification in classical molecular dynamics

“…In some instances, prediction errors due to model inadequacy can be handled by statistical correction of predictions, which may provide a reliable uncertainty measure [20]. Various surrogate methods have been developed for the estimation of prediction uncertainty, such as bootstrap-based methods, Gaussian process regression, neural networks and deep learning ensembles [21–23]. Gaussian process regression has been employed to identify particular calculations within a given dataset for which the uncertainties exceed a given threshold [24,25].…”

Fitting elephants in the density functionals zoo: Statistical criteria for the evaluation of density functional theory methods as a suitable replacement for counting parameters

“…This lack of correlation supports the main message of this work: The number of fitted parameters does not represent an effective measure of the transferability of a functional. More reliable statistical criteria—such as those developed in this work, or alternatively, the probabilistic performance estimator recently introduced by Pernot and Savin [ 91,92 ] —should be used to evaluate the reliability of new and existing xc functionals.…”

Using the Gini coefficient to characterize the shape of computational chemistry error distributions

“…We can relate this to an increasing trend of the errors with the bandgap value. 4 In the case of PER2018, the error distributions present also large skewness and kurtosis, that can be associated with the chemical heterogeneity of the dataset. 2 For THA2015, it was noted previously 35,4 that some experimental reference data with large measurement uncertainty could not be reproduced by any method in the studied set.…”

“…As a benchmarking statistic, the popular mean unsigned error (MUE) bears no information on such a risk. [1][2][3][4] We have recently reported a case where two unbiased error distributions with identical values of the MUE present widely different risks of large errors because of heavy tails in one of them. 5,4 It would therefore be very useful to complement the MUE with a statistic indicating or quantifying the risk of large errors.…”

“…13 The Lorenz curve has many mathematical representations, the most interesting one, for us, being its formulation as an integral of the quantile function, a direct link with our study of probabilistic metrics. [2][3][4] More precisely, we explore here the interest of the Gini coefficient as a complement to the MUE in benchmarking studies.…”