Benchmarking Scalable Predictive Uncertainty in Text Classification

Landeghem, Jordy Van; Blaschko, Matthew B.; Anckaert, Bertrand; Moens, Marie‐Francine

doi:10.1109/access.2022.3168734

Cited by 8 publications

(2 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Landeghem et al proposed a combination of deep ensemble and concrete dropout to model predictive uncertainty in natural language processing, specifically in multiclass and multilabel text classification tasks. The proposed method shows superior performance in calibration on i.i.d data, cross-domain classification, and novel class robustness [24]. Rodríguez-Puigvert et al apply MC-Dropout in all layers of the DCNN-based encoder to produce better uncertainty quantification for robotic perception.…”

Section: ) Recent Bdl Applicationsmentioning

confidence: 99%

On Performance and Calibration of Natural Gradient Langevin Dynamics

et al. 2023

View full text Add to dashboard Cite

Producing deep neural network (DNN) models with calibrated confidence is essential for applications in many fields, such as medical image analysis, natural language processing, and robotics. Modern neural networks have been reported to be poorly calibrated compared with those from a decade ago. The stochastic gradient Langevin dynamics (SGLD) algorithm offers a tractable approximate Bayesian inference applicable to DNN, providing a principled method for learning the uncertainty. A recent benchmark study showed that SGLD could produce a more robust model to covariate shifts than other competing methods. However, vanilla SGLD is also known to be slow, and preconditioning can improve SGLD efficacy. This paper proposes eigenvalue-corrected Kronecker factorization (EKFAC) preconditioned SGLD (EKS-GLD), in which a novel second-order gradient approximation is employed as a preconditioner for the SGLD algorithm. This approach is expected to bring together the advantages of both second-order optimization and the approximate Bayesian method. Experiments were conducted to compare the performance of EKSGLD with existing preconditioning methods and showed that it could achieve higher predictive accuracy and better calibration on the validation set. EKSGLD improved the best accuracy by 3.06% on CIFAR-10 and 4.15% on MNIST, improved the best negative log-likelihood by 16.2% on CIFAR-10 and 11.4% on MNIST, and improved the best thresholded adaptive calibration error by 4.05% on CIFAR-10.

show abstract

Section: ) Recent Bdl Applicationsmentioning

confidence: 99%

On Performance and Calibration of Natural Gradient Langevin Dynamics

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Lastly, due to the way that uncertainty estimates are evaluated, investigating distributional shift in NLP is also of interest, for instance through the work of Arora et al (2021), Kamath et al (2020), who focus on question answering and Tan et al (2019) for text classification. The most similar work to ours is the text classification uncertainty benchmark by Van Landeghem et al (2022), however they do not consider the impact of data or language, and test a different selection of models.…”

Section: Introductionmentioning

confidence: 99%

Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity

Ulmer¹,

Frellsen²,

Hardmeier³

2022

Findings of the Association for Computational Linguistics: EMNLP 2022

View full text Add to dashboard Cite

We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are opensourced in a software package. 1 The model zoo is available under https://github.com/ Kaleidophon/nlp-uncertainty-zoo, with the code for the experiments available under https://github.com/ Kaleidophon/nlp-low-resource-uncertainty.2 That is, unless the model class we chose is too restrictive.

show abstract