Systematic Investigation of Error Distribution in Machine Learning Algorithms Applied to the Quantum-Chemistry QM9 Data Set Using the Bias and Variance Decomposition

Abstract

Most machine learning applications in quantumchemistry (QC) data sets rely on a single statistical error parameter such as the mean square error (MSE) to evaluate their performance. However, this approach has limitations or can even yield incorrect interpretations. Here, we report a systematic investigation of the two components of the MSE, i.e., the bias and variance, using the QM9 data set. To this end, we experiment with three descriptors, namely (i) symmetry functions (SF, with two-body and three-body functions), (ii) many-body tensor representation (MBTR, with two-and three-body terms), and (iii) smooth overlap of atomic positions (SOAP), to evaluate the prediction process's performance using different numbers of molecules in training samples and the effect of bias and variance on the final MSE. Overall, low sample sizes are related to higher MSE. Moreover, the bias component strongly influences the larger MSEs. Furthermore, there is little agreement among molecules with higher errors (outliers) across different descriptors. However, there is a high prevalence among the outliers intersection set and the convex hull volume of geometric coordinates (VCH). According to the obtained results with the distribution of MSE (and its components bias and variance) and the appearance of outliers, it is suggested to use ensembles of models with a low bias to minimize the MSE, more specifically when using a small number of molecules in the training set. Article pubs.acs.org/jcim

SMICLR: Contrastive Learning on Multiple Molecular Representations for Semisupervised and Unsupervised Representation Learning

“…In our computational experiments, we explore the QM9 and MUTAG , data sets. QM9 is a popular data set used as a benchmark in several ML algorithms for quantum chemistry applications. ,, It comprises 134,000 organic molecules composed of carbon (C), hydrogen (H), oxygen (O), nitrogen (N), and fluorine (F) with molecules containing at maximum nine heavy atoms. It also reports 15 molecular properties (e.g., geometric, energetic, electronic, and thermodynamic properties) computed with the density functional theory (DFT) B3LYP/6-31G­(2 df , p ) framework.…”

GB‐score: Minimally designed machine learning scoring function based on distance‐weighted interatomic contact features

“…One proposed explanation for this undermining can be related to the unavailability of docking poses in the training set [51]. The second important problem is considering better assessment of ML-based scoring functions through uncertainty quantification [52][53] and domains of applicability [54] because all newly proposed scoring functions are converging to the same performance, which makes them indistinguishable. So analyzing model test error, its uncertainty across test sets, we can spot sub-domains in which different models perform better than the others and practice this assessment for further model comparison.…”

An interpretable stacking ensemble learning framework based on multi-dimensional data for real-time prediction of drug concentration: The example of olanzapine

“…Aside from the MAE, the following evaluation metrics were used to quantitatively evaluate the performance of the models: R-square ( ), mean squared error (MSE) ( Cesar de Azevedo et al, 2021 ), mean relative error (MRE) (%) ( Zhu et al, 2021a ), and ideal rate (IR) (%) ( Guo et al, 2021 ). These indices were calculated as follows: where , , and are the predicted, measured values, and the mean values, respectively.…”