Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology

Abstract

Trust and transparency are critical for deploying deep learning (DL) models into the clinic. DL application poses generalisation obstacles since training/development datasets often have different data distributions to clinical/production datasets that can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models used to predict cancer of unknown primary with three independent RNA-seq datasets covering 10,968 samples across 57 primary cancer types. Our results highlight simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation (e.g., p- value = 0.0013 for calibration). Moreover, we demonstrate Bayesian DL substantially improves accuracy under data distributional shifts when utilising "uncertainty thresholding" by designing a prototypical metric that evaluates the expected (accuracy) loss when deploying models from development to production, which we call the Area between Development and Production curve (ADP). In summary, Bayesian DL is a hopeful avenue of research for generalising uncertainty, which improves performance, transparency, and therefore safety of DL models for deployment in real-world.

STimage:robust, confident and interpretable models for predicting gene markers from cancer histopathological images

“…To increase the model robustness for a broad range of diverse, unseen datasets, we trained an ensemble of STimage regression models, which have been shown to improve generalisation in the transcriptomics context [21]. In addition to improving the robustness, ensembles allow us to quantify uncertainty in terms of the data (i.e., aleatoric uncertainty), as well as uncertainty about the model itself (i.e., epistemic uncertainty).…”