Quantifying sources of uncertainty in drug discovery predictions with probabilistic models

Lazic, Stanley E.; Williams, Dominic P.

doi:10.1016/j.ailsci.2021.100004

Cited by 7 publications

(8 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, the median value of all compounds in the generated dataset defines the threshold, but the range of allowed thresholds are fixed to be 4 ≤ pValue ≤ 6, where pValue = − log 10 (Activity Value). If the median is outside this range, a fixed threshold pValue = 5 is applied, which follows the common practice (Mayr et al, 2018;Stanley et al, 2021) in drug discovery. In this way, we can try our best to keep the dataset balanced while making the generated tasks meaningful.…”

Section: Binary Classification Task With Adaptive Thresholdmentioning

confidence: 99%

“…Additionally, current OOD research focus almost exclusively on the single-instance prediction tasks while overlook multi-instance prediction tasks, and how to better handle the distribution shift in multiple instance domains (e.g., molecule and protein inputs in the SBAP task) remains an open problem. Lastly, while not explored in this paper, the large-scale realistic datasets always come with non-negligible inherent noise, both aleatoric and epistemic (Lazic & Williams, 2021). And how to incorporate noise learning with OOD generalization to boost model's robustness and generalization in the meantime is an important research direction.…”

Section: Domain Generalization Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

Ji¹,

Zhang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

AI-aided drug discovery (AIDD) is gaining increasing popularity due to its promise of making the search for new pharmaceuticals quicker, cheaper and more efficient. In spite of its extensive use in many fields, such as ADMET prediction, virtual screening, protein folding and generative chemistry, little has been explored in terms of the out-ofdistribution (OOD) learning problem with noise, which is inevitable in real world AIDD applications.In this work, we present DrugOOD 1 , a systematic OOD dataset curator and benchmark for AI-aided drug discovery, which comes with an open-source Python package that fully automates the data curation and OOD benchmarking processes. We focus on one of the most crucial problems in AIDD: drug target binding affinity prediction, which involves both macromolecule (protein target) and small-molecule (drug compound). In contrast to only providing fixed datasets, DrugOOD offers automated dataset curator with user-friendly customization scripts, rich domain annotations aligned with biochemistry knowledge, realistic noise annotations and rigorous benchmarking of state-of-the-art OOD algorithms. Since the molecular data is often modeled as irregular graphs using graph neural network (GNN) backbones, DrugOOD also serves as a valuable testbed for graph OOD learning problems. Extensive empirical studies have shown a significant performance gap between in-distribution and out-of-distribution experiments, which highlights the need to develop better schemes that can allow for OOD generalization under noise for AIDD.

show abstract

Section: Binary Classification Task With Adaptive Thresholdmentioning

confidence: 99%

Section: Domain Generalization Algorithmsmentioning

confidence: 99%

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

Ji¹,

Zhang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Although there are ML methods that provide inherent prediction uncertainties such as Gaussian process modeling [ 20 ], most ML/DL methods including DNNs produce numerical end points without uncertainty estimates. Approaches for uncertainty quantification of predictions complementing ML/DL include probabilistic [ 21 ] and ensemble methods [ 22 ]. The latter methods determine prediction variance on the basis of differently trained models generated with the same algorithm.…”

Section: Quantifying Prediction Uncertaintymentioning

confidence: 99%

“…The latter methods determine prediction variance on the basis of differently trained models generated with the same algorithm. For probabilistic approaches, Bayesian DNNs provide a prime example [ 21 ]. However, due to their computational costs, Bayesian DNNs are only applicable to large data sets if approximations are introduced.…”

Section: Quantifying Prediction Uncertaintymentioning

confidence: 99%

Artificial Intelligence in Interdisciplinary Life Science and Drug Discovery Research

Bajorath

2022

Future Sci. OA

View full text Add to dashboard Cite

“…For this to be realised, a statistical methodology must be adopted through which the outputs of the various approaches can be integrated to produce updated judgments. Bayesian inference represents a powerful technique for achieving this, permitting as it does the generation of probabilistic distributions which may be related to severity of toxicity (Lazic and Williams 2021 ). Its application within predictive toxicology constitutes an emerging field of interest, and as such it has been drawn upon in recent studies aimed towards development of models describing endpoints spanning skin sensitivity (Reynolds et al 2019 ), drug-induced liver injury (Semenova et al 2020 ; Williams et al 2020 ) and cardiotoxicity (Felli and Leishman 2020 ).…”

Section: Introductionmentioning

confidence: 99%

The use of Bayesian methodology in the development and validation of a tiered assessment approach towards prediction of rat acute oral toxicity

et al. 2022

View full text Add to dashboard Cite

There exists consensus that the traditional means by which safety of chemicals is assessed—namely through reliance upon apical outcomes obtained following in vivo testing—is increasingly unfit for purpose. Whilst efforts in development of suitable alternatives continue, few have achieved levels of robustness required for regulatory acceptance. An array of “new approach methodologies” (NAM) for determining toxic effect, spanning in vitro and in silico spheres, have by now emerged. It has been suggested, intuitively, that combining data obtained from across these sources might serve to enhance overall confidence in derived judgment. This concept may be formalised in the “tiered assessment” approach, whereby evidence gathered through a sequential NAM testing strategy is exploited so to infer the properties of a compound of interest. Our intention has been to provide an illustration of how such a scheme might be developed and applied within a practical setting—adopting for this purpose the endpoint of rat acute oral lethality. Bayesian statistical inference is drawn upon to enable quantification of degree of confidence that a substance might ultimately belong to one of five LD50-associated toxicity categories. Informing this is evidence acquired both from existing in silico and in vitro resources, alongside a purposely-constructed random forest model and structural alert set. Results indicate that the combination of in silico methodologies provides moderately conservative estimations of hazard, conducive for application in safety assessment, and for which levels of certainty are defined. Accordingly, scope for potential extension of approach to further toxicological endpoints is demonstrated.

show abstract

Quantifying sources of uncertainty in drug discovery predictions with probabilistic models

Cited by 7 publications

References 37 publications

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

Artificial Intelligence in Interdisciplinary Life Science and Drug Discovery Research

The use of Bayesian methodology in the development and validation of a tiered assessment approach towards prediction of rat acute oral toxicity

Contact Info

Product

Resources

About