Towards Reducing Aleatoric Uncertainty for Medical Imaging Tasks

Sambyal, Abhishek Singh; Krishnan, Narayanan C.; Bathula, Deepti R.

doi:10.48550/arxiv.2110.11012

Cited by 4 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Epistemic and aleatoric uncertainties present the potential for further insights, including whether a data point's predictive uncertainty will reduce with either more examples or by an altered model design (epistemic uncertainty), or more features (aleatoric uncertainty) [51]- [54].…”

Section: Discussionmentioning

confidence: 99%

“…Aleatoric uncertainty is dependent on data’s inherent noise and can be reduced with more data features that explain variance caused by confounding variables (e.g., patient age, cancer stage, batch effect). Epistemic and aleatoric uncertainties present the potential for further insights, including whether a data point’s predictive uncertainty will reduce with either more examples or by an altered model design (epistemic uncertainty), or more features (aleatoric uncertainty) [51]–[54].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

MacDonald

Foley²,

Yap³

et al. 2022

Preprint

View full text Add to dashboard Cite

Trust and transparency are critical for deploying deep learning (DL) models into the clinic. DL application poses generalisation obstacles since training/development datasets often have different data distributions to clinical/production datasets that can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models used to predict cancer of unknown primary with three independent RNA-seq datasets covering 10,968 samples across 57 primary cancer types. Our results highlight simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation (e.g., p- value = 0.0013 for calibration). Moreover, we demonstrate Bayesian DL substantially improves accuracy under data distributional shifts when utilising "uncertainty thresholding" by designing a prototypical metric that evaluates the expected (accuracy) loss when deploying models from development to production, which we call the Area between Development and Production curve (ADP). In summary, Bayesian DL is a hopeful avenue of research for generalising uncertainty, which improves performance, transparency, and therefore safety of DL models for deployment in real-world.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

MacDonald

Foley²,

Yap³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Although the aleatoric uncertainty is supposed to be irreducible for a specific dataset, incorporating additional features or improving the quality of the existing features can assist in its reduction [36] In the latter, the goal is to retrieve additional evidence that supports or contradicts a given hypothesis. In the former, it is the case of classification with rejection, which is a viable option, where the presence and cost of errors can be detrimental to the performance of automated classification systems [33].…”

Section: Methodsmentioning

confidence: 99%

Uncertainty-Based Rejection in Machine Learning: Implications for Model Development and Interpretability

et al. 2022

View full text Add to dashboard Cite

Uncertainty is present in every single prediction of Machine Learning (ML) models. Uncertainty Quantification (UQ) is arguably relevant, in particular for safety-critical applications. Prior research focused on the development of methods to quantify uncertainty; however, less attention has been given to how to leverage the knowledge of uncertainty in the process of model development. This work focused on applying UQ into practice, closing the gap of its utility in the ML pipeline and giving insights into how UQ is used to improve model development and its interpretability. We identified three main research questions: (1) How can UQ contribute to choosing the most suitable model for a given classification task? (2) Can UQ be used to combine different models in a principled manner? (3) Can visualization techniques improve UQ’s interpretability? These questions are answered by applying several methods to quantify uncertainty in both a simulated dataset and a real-world dataset of Human Activity Recognition (HAR). Our results showed that uncertainty quantification can increase model robustness and interpretability.

show abstract

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

MacDonald

Foley²,

Yap³

et al. 2023

Sci Rep

View full text Add to dashboard Cite

Uncertainty estimation is crucial for understanding the reliability of deep learning (DL) predictions, and critical for deploying DL in the clinic. Differences between training and production datasets can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models for predicting cancer of unknown primary, using three RNA-seq datasets with 10,968 samples across 57 cancer types. Our results highlight that simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation. Moreover, we designed a prototypical metric—the area between development and production curve (ADP), which evaluates the accuracy loss when deploying models from development to production. Using ADP, we demonstrate that Bayesian DL improves accuracy under data distributional shifts when utilising ‘uncertainty thresholding’. In summary, Bayesian DL is a promising approach for generalising uncertainty, improving performance, transparency, and safety of DL models for deployment in the real world.

show abstract

Towards Reducing Aleatoric Uncertainty for Medical Imaging Tasks

Cited by 4 publications

References 0 publications

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

Uncertainty-Based Rejection in Machine Learning: Implications for Model Development and Interpretability

Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

Contact Info

Product

Resources

About