Comparing Multiple Machine Learning Algorithms and Metrics for Estrogen Receptor Binding Prediction

Abstract

Many chemicals that disrupt endocrine function have been linked to a variety of adverse biological outcomes. However, screening for endocrine disruption using in vitro or in vivo approaches is costly and time-consuming. Computational methods, e.g., quantitative structure-activity relationship models, have become more reliable due to bigger training sets, increased computing power, and advanced machine learning algorithms, such as multilayered artificial neural networks. Machine learning models can be used to predict compounds for endocrine disrupting capabilities, such as binding to the estrogen receptor (ER), and allow for prioritization and further testing. In this work, an exhaustive comparison of multiple machine learning algorithms, chemical spaces, and evaluation metrics for ER binding was performed on public data sets curated using in-house cheminformatics software (Assay Central). Chemical features utilized in modeling consisted of binary fingerprints (ECFP6, FCFP6, ToxPrint, or MACCS keys) and continuous molecular descriptors from RDKit. Each feature set was subjected to classic machine learning algorithms (Bernoulli Naive Bayes, AdaBoost Decision Tree, Random Forest, Support Vector Machine) and Deep Neural Networks (DNN). Models were evaluated using a variety of metrics: recall, precision, F1-score, accuracy, area under the receiver operating characteristic curve, Cohen's Kappa, and Matthews correlation coefficient. For predicting compounds within the training set, DNN has an accuracy higher than that of other methods; however, in 5-fold cross validation and external test set predictions, DNN and most classic machine learning models perform similarly regardless of the data set or molecular descriptors used. We have also used the rank normalized scores as a performance-criteria for each machine learning method, and Random Forest performed best on the validation set when ranked by metric or by data sets. These results suggest classic machine learning algorithms may be sufficient to develop high quality predictive models of ER activity.

Towards accurate high-throughput ligand affinity prediction by exploiting structural ensembles, docking metrics and ligand similarity

“…This led to a prediction tool outperforming methods based either solely on SBVS or LBVS approaches as exemplified here with the MACCS fingerprints. Our work confirmed the performance of Random Forest over other machine learning approaches as previously noticed (Russo et al , 2018). In some cases, higher accuracy was reported but for smaller compound libraries (Hou et al , 2018).…”

Repurposing Pyramax® for the Treatment of Ebola Virus Disease: Additivity of the Lysosomotropic Pyronaridine and Non-Lysosomotropic Artesunate

“…The Assay Central software has been previously described (23)(24)(25)(26)(27)(28)(29)(30)(31)(32) which uses the source code management system Git to gather and store structure-activity datasets collated in Molecular Notebook (Molecular Materials Informatics, Inc. in Montreal, Canada). The output is a high-quality dataset and a Bayesian model using extendedconnectivity fingerprints of maximum diameter 6 (ECFP6) descriptors.…”

“…The output is a high-quality dataset and a Bayesian model using extendedconnectivity fingerprints of maximum diameter 6 (ECFP6) descriptors. Each model includes several metrics to evaluate and compare predictive performance as previously described in a relevant publication (29), including Receiver Operator Characteristic, 6 Recall, Precision, F1 Score, Cohen's Kappa (33,34), and Matthews Correlation Coefficient (35). Applicability is representative of the overlap between the training and the test set.…”

Capsule Networks Showed Excellent Performance in the Classification of hERG Blockers/Nonblockers

“…They have widely been applied to fields particularly computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, bioinformatics, and various games (Collobert and Weston, 2008;Bengio, 2009;Dahl et al, 2012;Hinton et al, 2012;LeCun et al, 2015;Defferrard et al, 2016;Mamoshina et al, 2016), where they have produced results comparable to or in some cases superior to human experts. In recent years, deep learning has also been applied to drug discovery, and it has demonstrated its potentials (Lusci et al, 2013;Ma et al, 2015;Xu et al, 2015;Aliper et al, 2016;Mayr et al, 2016;Pereira et al, 2016;Subramanian et al, 2016;Kadurin et al, 2017;Ragoza et al, 2017;Ramsundar et al, 2017;Xu et al, 2017;Ghasemi et al, 2018;Harel and Radinsky, 2018;Hu et al, 2018;Popova et al, 2018;Preuer et al, 2018;Russo et al, 2018;Segler et al, 2018;Shin et al, 2018;Cai et al, 2019;Wang et al, 2019a;Yang et al, 2019). However, there are still some issues that limit the application of deep learning in drug discovery.…”