Error Tolerance of Machine Learning Algorithms across Contemporary Biological Targets

Abstract

Machine learning continues to make strident advances in the prediction of desired properties concerning drug development. Problematically, the efficacy of machine learning in these arenas is reliant upon highly accurate and abundant data. These two limitations, high accuracy and abundance, are often taken together; however, insight into the dataset accuracy limitation of contemporary machine learning algorithms may yield insight into whether non-bench experimental sources of data may be used to generate useful machine learning models where there is a paucity of experimental data. We took highly accurate data across six kinase types, one GPCR, one polymerase, a human protease, and HIV protease, and intentionally introduced error at varying population proportions in the datasets for each target. With the generated error in the data, we explored how the retrospective accuracy of a Naïve Bayes Network, a Random Forest Model, and a Probabilistic Neural Network model decayed as a function of error. Additionally, we explored the ability of a training dataset with an error profile resembling that produced by the Free Energy Perturbation method (FEP+) to generate machine learning models with useful retrospective capabilities. The categorical error tolerance was quite high for a Naïve Bayes Network algorithm averaging 39% error in the training set required to lose predictivity on the test set. Additionally, a Random Forest tolerated a significant degree of categorical error introduced into the training set with an average error of 29% required to lose predictivity. However, we found the Probabilistic Neural Network algorithm did not tolerate as much categorical error requiring an average of 20% error to lose predictivity. Finally, we found that a Naïve Bayes Network and a Random Forest could both use datasets with an error profile resembling that of FEP+. This work demonstrates that computational methods of known error distribution like FEP+ may be useful in generating machine learning models not based on extensive and expensive in vitro-generated datasets.

Prospective evaluation and success of a machine learning hit-to-lead drug development program against phosphatidylinositol 3-kinase α

“…29,30 Additionally, the Bayesian classifier has the additional property of being highly noise tolerant as compared to other machine learning methods. 31 Another advantage of using the Bayesian classifier applied in FRESH is that it uses 2D extended connectivity fingerprints (ECFP) making it suitable for use on large data sets due to its rapid speed. 32 The Bayesian classifier is therefore applied ahead of more computationally expensive methods like docking and binding free energy calculations in the task-stream of FRESH.…”

Machine Learning Methods in Drug Discovery

“…Information for the particular protein of interest might be limited, resulting in not much-extrapolated data. Free Energy Perturbation method is a platform where biological information regarding the protein is generated based on computational screening [ 61 ]. Data gathered from this method is utilized for training algorithms; however, not all the information is collected from a wet lab, rather computer-generated prediction is utilized.…”

“…The accuracy of the training data might be lower than anticipated. Even though algorithms discussed in this review have a higher threshold for minimizing errors, there are still some categorical errors from training sets [ 61 ].…”

Discovering the arrow of time in machine learning

“…Nevertheless, there are trade-offs as unrepresentative data can lead to 'over-fitting' and poor generalisability, which is when a ML algorithm performs very well on the training data but poorly on other data. ML is also typically understood to be tolerant of errors in the data, with some approaches showing acceptable performance despite error rates up to 39 percent [1,2]. ML methods usually require substantial calculation, but this can be managed through careful selection of algorithms and setups [3].…”

“…Broadly, ML uses computational methods and algorithms to learn to perform tasks, such as categorisation, decision making or anomaly detection through experience and without explicit instruction. ML is most effective in situations where non-computational means or conventional algorithms are impractical or impossible, such as when the data are vast, complex, highly variable and/or full of errors [1,2]. Thus, ML is useful for analysing natural language, images, or other types of complex and messy data that are now available in ever-growing and impractically large volumes.…”