Blinded Predictions and Post Hoc Analysis of the Second Solubility Challenge Data: Exploring Training Data and Feature Set Selection for Machine and Deep Learning Models

Abstract

Accurate methods to predict solubility from molecular structure are highly sought after in the chemical sciences. To assess the state of the art, the American Chemical Society organized a “Second Solubility Challenge” in 2019, in which competitors were invited to submit blinded predictions of the solubilities of 132 drug-like molecules. In the first part of this article, we describe the development of two models that were submitted to the Blind Challenge in 2019 but which have not previously been reported. These models were based on computationally inexpensive molecular descriptors and traditional machine learning algorithms and were trained on a relatively small data set of 300 molecules. In the second part of the article, to test the hypothesis that predictions would improve with more advanced algorithms and higher volumes of training data, we compare these original predictions with those made after the deadline using deep learning models trained on larger solubility data sets consisting of 2999 and 5697 molecules. The results show that there are several algorithms that are able to obtain near state-of-the-art performance on the solubility challenge data sets, with the best model, a graph convolutional neural network, resulting in an RMSE of 0.86 log units. Critical analysis of the models reveals systematic differences between the performance of models using certain feature sets and training data sets. The results suggest that careful selection of high quality training data from relevant regions of chemical space is critical for prediction accuracy but that other methodological issues remain problematic for machine learning solubility models, such as the difficulty in modeling complex chemical spaces from sparse training data sets.

Chemical space analysis and property prediction for carbon capture solvent molecules

“…5 displays the chemical space graphically and follows the protocol described in some of the author's previous work. 89 This figure is generated using the zinc-20938 and ccs-lit-167 data sets. In this figure each molecule is represented as a node in the graph and the most similar (Tanimoto similarity scores of ≥0.7 using Morgan fingerprints with a radius of 2 and 2048 bits) are connected.…”

Evaluating Machine Learning Methods of Analyzing Multiclass Metabolomics

“…For each classification method, 50% of the samples were chosen as the training set by random sampling. The remaining samples were regarded as the test set . As shown in Figure , ROC plots were obtained using the (A) AdaBoost, (B) bagging, (C) DT, (D) KNN, (E) LDA, (F) NB, (G) PLS, (H) RF, and (I) SVM methods.…”

“…The remaining samples were regarded as the test set. 97 As shown in Figure 5, ROC plots were obtained using the (A) AdaBoost, (B) bagging, (C) DT, (D) KNN, (E) LDA, (F) NB, (G) PLS, (H) RF, and (I) SVM methods. By comparison, the AUC values were significantly different when using different classification methods.…”

Free Energy Perturbation Approach for Accurate Crystalline Aqueous Solubility Predictions

“…We carry out a head-to-head comparison of the FEP+ solubility approach against state-of-the-art ML approaches using QM descriptors and show that the FEP+ approach achieves improved predictive accuracies (RMSE = 0.87 and 0.86 vs RMSE = 1.02 and 1.24) and differentiation power ( R 2 = 0.64 and 0.69 vs 0.40 and 0.25) for the Literature and AbbVie data sets, respectively. While recent developments in neural network models have achieved similar accuracies in the recent solubility blind test ( R 2 = 0.54, RMSE = 0.86) on a large set of chemically diverse compounds, the performance of such models may vary significantly depending on the data sets used for training …”

“…While recent developments in neural network models have achieved similar accuracies in the recent solubility blind test (R 2 = 0.54, RMSE = 0.86) on a large set of chemically diverse compounds, the performance of such models may vary significantly depending on the data sets used for training. 36 In addition to the predictive power of the crystalline FEP+ approach for both structurally similar and diverse compounds, the method also provides unique energetic insights by dissecting the underlying contributions of the solubility into sublimation and hydration energy components. Such insights can reveal invaluable information for rational drug design, as well as guide tailored formulation approaches for drug development.…”