A comparison of Monte Carlo dropout and bootstrap aggregation on the performance and uncertainty estimation in radiation therapy dose prediction with deep learning neural networks

Abstract

Recently, artificial intelligence technologies and algorithms have become a major focus for advancements in treatment planning for radiation therapy. As these are starting to become incorporated into the clinical workflow, a major concern from clinicians is not whether the model is accurate, but whether the model can express to a human operator when it does not know if its answer is correct. We propose to use Monte Carlo Dropout (MCDO) and the bootstrap aggregation (bagging) technique on deep learning (DL) models to produce uncertainty estimations for radiation therapy dose prediction. We show that both models are capable of generating a reasonable uncertainty map, and, with our proposed scaling technique, creating interpretable uncertainties and bounds on the prediction and any relevant metrics. Performance-wise, bagging provides statistically significant reduced loss value and errors in most of the metrics investigated in this study. The addition of bagging was able to further reduce errors by another 0.34% for D m e a n and 0.19% for D max , on average, when compared to the baseline model. Overall, the bagging framework provided significantly lower mean absolute error (MAE) of 2.62, as opposed to the baseline model’s MAE of 2.87. The usefulness of bagging, from solely a performance standpoint, does highly depend on the problem and the acceptable predictive error, and its high upfront computational cost during training should be factored in to deciding whether it is advantageous to use it. In terms of deployment with uncertainty estimations turned on, both methods offer the same performance time of about 12 s. As an ensemble-based metaheuristic, bagging can be used with existing machine learning architectures to improve stability and performance, and MCDO can be applied to any DL models that have dropout as part of their architecture.

Estimating uncertainty in radiation oncology dose prediction with dropout and bootstrap in U-Net models

“…In fact, a large standard deviation means that the model predicts different values for the same input and can be considered uncertain. The formulas that have been proposed in [7] are…”

Probabilistic feature extraction, dose statistic prediction and dose mimicking for automated radiation therapy treatment planning

“…While certain aspects of a general probabilistic approach have been previously addressedfor instance, in Nilsson et al [29], where a U-net-based mixture density network is trained to output voxelwise univariate Gaussian mixtures, or in Nguyen et al [30], where ensemble and Monte Carlo dropout techniques are used to provide uncertainties of spatial dose and dose statistics by sampling predictions-the estimation of concrete predictive distributions for DVHs or other dose statistics has yet to be given much attention in literature. In particular, given a collection of dose statistics, which may represent a discretized DVH, one would like to estimate their multivariate conditional probability distribution given the current patient geometry and the training dataset.…”

“…Thus, a spatial dose prediction algorithm may complement the current dose mimicking problem setup by helping to shape the dose distribution outside the defined ROIs. If done also probabilistically, such as in Nguyen et al [30], the spatial dose prediction could be basis for a contribution to the dose mimicking objective in (2) penalizing spatial deviation, similarly to in Nilsson et al [29], as well as serve as a reference for our proposed dose statistic prediction method.…”

Probabilistic Pareto plan generation for semiautomated multicriteria radiation therapy treatment planning

“…but although some previous work (Nguyen et al 2021, Nilsson et al 2021 has been devoted to uncertainty estimation for spatial dose prediction, no existing method is able to output the complete multivariate predictive distribution as a closed-form probability density. Instead, we will proceed with a deterministic U-net dose prediction model f : X → D and an associated regression problem on the form…”

“…In this context, we showed ) that probabilistic methods, which output predictive probability distributions expressing estimation uncertainties, may reduce the information loss between the prediction and mimicking parts. Indeed, much of other previous work (Covele et al 2021, Fogliata et al 2019, Nguyen et al 2021, Nilsson et al 2021 have already been directed toward precise quantification of predictive uncertainties for spatial dose or DVH statistics.…”