Using the Gini coefficient to characterize the shape of computational chemistry error distributions

Pernot, Pascal; Savin, Andreas

doi:10.1007/s00214-021-02725-0

Cited by 9 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The distribution of errors in density functional approximations is often not normal [5]. This can be seen in Figure 9.…”

Section: Eliminating Strange Results?mentioning

confidence: 98%

“…Although these numbers convey some information, they sometimes hide the misconception that the distribution of errors is normal. In the cases we analyze below, as in most cases we studied previously [5], the distributions of errors are not normal. This justifies the use of probabilistic estimators, such as those presented in our previous work [1][2][3], or the ones introduced here.…”

Section: Statistical Measuresmentioning

confidence: 99%

See 1 more Smart Citation

Should We Gain Confidence from the Similarity of Results between Methods?

Pernot

Savin

2022

Computation

Self Cite

View full text Add to dashboard Cite

Confirming the result of a calculation by a calculation with a different method is often seen as a validity check. However, when the methods considered are all subject to the same (systematic) errors, this practice fails. Using a statistical approach, we define measures for reliability and similarity, and we explore the extent to which the similarity of results can help improve our judgment of the validity of data. This method is illustrated on synthetic data and applied to two benchmark datasets extracted from the literature: band gaps of solids estimated by various density functional approximations, and effective atomization energies estimated by ab initio and machine-learning methods. Depending on the levels of bias and correlation of the datasets, we found that similarity may provide a null-to-marginal improvement in reliability and was mostly effective in eliminating large errors.

show abstract

“…The distribution of errors in density functional approximations is often not normal [5]. This can be seen in Figure 9.…”

Section: Eliminating Strange Results?mentioning

confidence: 98%

Section: Statistical Measuresmentioning

confidence: 99%

Should We Gain Confidence from the Similarity of Results between Methods?

Pernot

Savin

2022

Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…Model errors can present any type of distribution 7 . However, a recent study on the shape of error distributions for different QoIs and DFAs showed that the distributions are most often unimodal, with various levels of asymmetry, and that they are in most cases heavier-tailed than a normal distribution 62 .…”

Section: Model Errorsmentioning

confidence: 99%

The long road to calibrated prediction uncertainty in computational chemistry

Pernot

2022

Preprint

Self Cite

View full text Add to dashboard Cite

a) Uncertainty quantification (UQ) in computational chemistry (CC) is still in its infancy.Very few CC methods are designed to provide a confidence level on their predictions, and most users still rely improperly on the mean absolute error as an accuracy metric. The development of reliable uncertainty quantification methods is essential, notably for computational chemistry to be used confidently in industrial processes. A review of the CC-UQ literature shows that there is no common standard procedure to report nor validate prediction uncertainty. I consider here analysis tools using concepts (calibration and sharpness) developed in meteorology and machine learning for the validation of probabilistic forecasters. These tools are adapted to CC-UQ and applied to datasets of prediction uncertainties provided by composite methods, Bayesian Ensembles methods, machine learning and a posteriori statistical methods.

show abstract

“…A contrario, it is not possible to design a prediction interval form a standard uncertainty u E i without making hypotheses on the error distribution. Being mostly dominated by model errors, computational chemistry error distributions are often non-normal, 25,26 which prevents the use of simple recipes (such as the 2σ rule). Making unsupported distribution hypotheses would add a fragility layer to the validation process, complicating the interpretation of negative validation tests.…”

Section: B Notationsmentioning

confidence: 99%

“…My goal was to derive a practical set of validation tools adapted to the specifics of CC-UQ, notably (i) the frequent use of statistical summaries (standard or expanded uncertainties), 23 instead of the prediction distributions expected by the CS framework, (ii) the possible presence of uncertainty on the reference data used for validation, (iii) the small size of most validation datasets when compared to ML applications, which limits the power of statistical validation tests, and (iv) the non-normality of the error distributions due to the frequent predominance of model errors. 9,[24][25][26] Considering these constraints, I was driven into considering two validation options, based on the available information. When expanded uncertainties are available, such as U 95 (the half-range of a 95% confidence interval), calibration should be tested by comparing the effective coverage of the corresponding prediction intervals to the target probability.…”

Section: Introductionmentioning

confidence: 99%

Prediction uncertainty validation for computational chemists

Pernot

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Validation of prediction uncertainty (PU) is becoming an essential tool for modern computational chemistry. Designed to quantify the reliability of predictions in meteorology, the calibration-sharpness (CS) framework is now widely used to optimize and validate uncertainty-aware machine learning (ML) methods. However, its application is not limited to ML and it can serve as a principled framework for any PU validation. The present article is intended as a step-by-step introduction to the concepts and techniques of PU validation in the CS framework, adapted to the specifics of computational chemistry. The presented methods range from elementary graphical checks to more sophisticated ones based on local calibration statistics. The concept of tightness, is introduced. The methods are illustrated on synthetic datasets and applied to uncertainty quantification data extracted from the computational chemistry literature.

show abstract

Using the Gini coefficient to characterize the shape of computational chemistry error distributions

Cited by 9 publications

References 38 publications

Should We Gain Confidence from the Similarity of Results between Methods?

Should We Gain Confidence from the Similarity of Results between Methods?

The long road to calibrated prediction uncertainty in computational chemistry

Prediction uncertainty validation for computational chemists

Contact Info

Product

Resources

About