Thinking Globally, Acting Locally: On the Issue of Training Set Imbalance and the Case for Local Machine Learning Models in Chemistry

Abstract

<div><div><div><p>The appropriate sampling of training data out of a potentially imbalanced data set is of critical importance for the development of robust and accurate machine learning models. A challenge that underpins this task is the partitioning of the data into groups of similar instances, and the analysis of the group populations. In molecular data sets, different groups of molecules may be hard to identify. However, if the distribution of a given data set is ignored then some of these groups may remain under-represented and the sampling biased, even if the size of data is large. In this study, we use the example of the Harvard Clean Energy Project (CEP) data set to assess the challenges posed by imbalanced data and the impact that accounting for different groups during the selection of training data has on the quality of the resulting machine learning models. We employ a partitioning criterion based on the underlying rules for the CEP molecular library generation to identify groups of structurally similar compounds. First, we evaluate the performance of regression models that are trained globally (i.e., by randomly sampling the entire data set for training data). This traditional approach serves as the benchmark reference. We compare its results with those of models that are trained locally, i.e., within each of the identified molecular domains. We demonstrate that local models outperform the best reported global models by considerable margins and are more efficient in their training data needs. We propose a strategy to redesign training sets for the development of improved global models. While the resulting uniform training sets can successfully yield robust global models, we identify the distribution mismatch between feature representations of different molecular domains as a critical limitation for any further improvement. We take advantage of the discovered distribution shift and propose an ensemble of classification and regression models to achieve a generalized and reliable model that outperforms the state-of-the-art model, trained on the CEP data set. Moreover, this study provides a benchmark for the development of future methodologies concerned with imbalanced chemical data.</p></div></div></div>

Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry

“…Chemical research is no longer an exception in this development [1], and numerous areas have been identified, in which ML is now employed to great effect (see, e.g., Refs. [2][3][4][5][6][7]). While ML applications have resulted in a number of exciting and valuable studies that have advanced chemical domain knowledge, it is worth noting that there is still a considerable lack of quality control, guidance, uniformity, and established protocols for the successful conduct of such studies.…”

A Physics-Infused Deep Learning Model for the Prediction of Refractive Indices and Its Use for the Large-Scale Screening of Organic Compound Space

“…For instance, we analyze the MAE of the predictions at the tail of the RI histogram, i.e., the desired remoter areas in the RI distribution. If the prediction error in those regions of the molecular candidates is worse than the average, we fine-tune predictive models so that they are able to capture the essence of those desired underrepresented class of molecules [50]. To perform fine-tuning (FT), we carefully retrain the best model on the data points that are close to the tail of the RI distribution, i.e., those that most probably deviate from their predicted values more than the overall MAE.…”

Leveraging Uncertainty from Deep Learning for Trustworthy Material Discovery Workflows

“…4,5 Existing efforts implicitly assume ideal conditions during both training and testing by assuming since a large amount of relevant historical data is rarely available and the test data is typically and systematically different from the training data either through noise 6 or other changes in the distribution. 7 This is an inherent challenge in applying DL to scientific domain whose aim is to find something "new" and "different" that outperforms existing materials. It is well known that DL models are highly susceptible to such distributional shifts, which often leads to unintended and potentially harmful behavior (i.e., overconfident on predictions), especially when trained with insufficient amount of data.…”

“…The majority of efforts in Material Informatics are currently devoted to training deep neural networks (DNNs) that can achieve high accuracy on holdout data set from the training distribution. , Existing efforts implicitly assume ideal conditions during both training and testing by assuming (a) having access to a sufficiently large labeled training data and (b) test data from the “same distribution” as the training set. Unfortunately, these conditions are seldomly met in Material Discovery applications since a large amount of relevant historical data is rarely available and the test data is typically and systematically different from the training data either through noise or other changes in the distribution . This is an inherent challenge in applying DL to scientific domain whose aim is to find something “new” and “different” that outperforms existing materials.…”