Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles

Sarawgi, Utkarsh; Zulfikar, Wazeer; Khincha, Rishab; Maes, Pattie

doi:10.48550/arxiv.2009.12406

Cited by 1 publication

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We further use the 65-95-99.7 rule (also called the empirical rule) to obtain calibration curves for a comprehensive analysis of calibration [17], [21]. To plot these curves, we first compute the x% prediction interval for each data point under evaluation based on Gaussian quantiles using the prediction value and variance.…”

Section: Resultsmentioning

confidence: 99%

“…Bayesian approximation techniques such as dropout-based VI [23], [24], expectation propagation [25], variational inference (VI) [26], [27], deterministic VI [28], neural networks as Gaussian processes [29], approximate Bayesian ensembling [30], and Bayesian model averaging in low-dimensional parameter subspaces [31] have been shown to be quite useful in modelling the uncertainties in neural networks. Non-Bayesian approaches [17], [21], [32]- [34] that involve bootstrapping and ensembling multiple probabilistic neural networks have shown performances comparable to Bayesian methods with reduced computational costs and modifications to the training procedure. Additionally, there is a breadth of other theoretical, empirical, and review works on estimating predictive uncertainties with neural networks [35]- [41].…”

Section: Related Work a Uncertainty Estimationmentioning

confidence: 99%

“…Chang et al [49] uses uncertainty estimates and prefers to learn the data points predicted incorrectly with higher uncertainty in different mini-batches of SGD. Some other works have addressed the advantages of incorporating uncertainty estimates in multi-modal settings [17]- [20].…”

Section: B Ensemble Techniquesmentioning

confidence: 99%

“…For uncertainty estimation, each of the models predicts a target distribution instead of a point estimate to account for the heteroscedasticity in data and yields predictive uncertainties along with the predicted mean value [17], [21], [40]. The target distribution is modelled as a Gaussian distribution p h j (y n |x j n ) parameterized by the mean µ h j and the standard deviation σ h j , predicted at the final layer of the models i.e.…”

Section: B Multi-modal Feature Extractionmentioning

confidence: 99%

“…Some works have briefly discussed uncertainty estimation in multi-modal settings [17]- [20]. We propose a notion of uncertainty-awareness with sequentially boosted ensembling in multi-modal settings.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Uncertainty-Aware Boosted Ensembling in Multi-Modal Settings

Sarawgi¹,

Khincha²,

Zulfikar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reliability of machine learning (ML) systems is crucial in safety-critical applications such as healthcare, and uncertainty estimation is a widely researched method to highlight the confidence of ML systems in deployment. Sequential and parallel ensemble techniques have shown improved performance of ML systems in multi-modal settings by leveraging the feature sets together. We propose an uncertainty-aware boosting technique for multi-modal ensembling in order to focus on the data points with higher associated uncertainty estimates, rather than the ones with higher loss values. We evaluate this method on healthcare tasks related to Dementia and Parkinson's disease which involve real-world multi-modal speech and text data, wherein our method shows an improved performance. Additional analysis suggests that introducing uncertainty-awareness into the boosted ensembles decreases the overall entropy of the system, making it more robust to heteroscedasticity in the data, as well as better calibrating each of the modalities along with high quality prediction intervals. We open-source our entire codebase at https://github.com/usarawgi911/Uncertainty-aware-boosting.

show abstract

Section: Resultsmentioning

confidence: 99%