Confidence Estimation via Auxiliary Models

Corbière, Charles; Thome, Nicolas; Saporta, Antoine; Vu, Tuan-Hung; Cord, Matthieu; Pérez, Patrick

doi:10.1109/tpami.2021.3085983

Cited by 39 publications

(18 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural language models, such as, e.g., models for machine translation, risk to be miscalibrated, too (Kumar and Sarawagi, 2019). Various remedies for miscalibration have been proposed and explored in the literature: modification of the loss-function (Moon et al, 2020); coupling and training of a complementary network to predict a prediction's reliability, that is, its empirical likelihood of being correct (Corbière et al, 2019); simultaneous training of an entire ensemble of deep neural networks (Lakshminarayanan et al, 2017); model distillation . Some pre-trained Transformers have, however, been claimed to be reasonably well calibrated.…”

Section: Related Work Accuracy and Consistency Of Nlms' Factual Kno...mentioning

confidence: 99%

Judgment aggregation, discursive dilemma and reflective equilibrium: Neural language models as self-improving doxastic agents

Betz

Richardson²

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Neural language models (NLMs) are susceptible to producing inconsistent output. This paper proposes a new diagnosis as well as a novel remedy for NLMs' incoherence. We train NLMs on synthetic text corpora that are created by simulating text production in a society. For diagnostic purposes, we explicitly model the individual belief systems of artificial agents (authors) who produce corpus texts. NLMs, trained on those texts, can be shown to aggregate the judgments of individual authors during pre-training according to sentence-wise vote ratios (roughly, reporting frequencies), which inevitably leads to so-called discursive dilemmas: aggregate judgments are inconsistent even though all individual belief states are consistent. As a remedy for such inconsistencies, we develop a self-training procedure—inspired by the concept of reflective equilibrium—that effectively reduces the extent of logical incoherence in a model's belief system, corrects global mis-confidence, and eventually allows the model to settle on a new, epistemically superior belief state. Thus, social choice theory helps to understand why NLMs are prone to produce inconsistencies; epistemology suggests how to get rid of them.

show abstract

Section: Related Work Accuracy and Consistency Of Nlms' Factual Kno...mentioning

confidence: 99%

Judgment aggregation, discursive dilemma and reflective equilibrium: Neural language models as self-improving doxastic agents

Betz

Richardson²

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…We first present the adaptation results on the task of GTAV → Cityscapes in TABLE 1, with comparisons to the baseline model [31] as well as the state-of-the-art DA approaches [35], [36], [38], [40], and the best results are highlighted in bold. Overall, our SePiCo (ProtoCL/BankCL/DistCL) yield a leading result among comparison methods [29], [34], [40], [62]. Particularly, we observe: (i) SePiCo (DistCL) achieves 61.0% mIoU, outperforming the baseline model trained merely on source data by a large margin of +22.4% mIoU; (ii) Adversarial training methods, e.g., AdaptSeg [20], CLAN [23], SIM [25], FADA [26], can improve the transferability, but the effect is not as obvious as using self-training methods, e.g., Seg-Uncert.…”

Section: Comparisons With the State-of-the-artsmentioning

confidence: 90%

“…Another line of work harnesses self-training to promote the segmentation performance [28], [29], [30], [31], [32], [33]. By adopting confidence estimation [34], consistency regularization [35], or label denoising [36], the bias in pseudo labels could be relieved to some extent. While many works are already capable of establishing milestone performance, there is still much room for improvement beyond the current state-of-the-art.…”

Section: Memory Bankmentioning

confidence: 99%

See 1 more Smart Citation

SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation

Xie¹,

Li²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Domain adaptive semantic segmentation attempts to make satisfactory dense predictions on an unlabeled target domain by utilizing the supervised model trained on a labeled source domain. One popular solution is self-training, which retrains the model with pseudo labels on target instances. Plenty of methods tend to alleviate noisy pseudo labels, however, they ignore intrinsic connections among cross-domain pixels with similar semantic concepts. In consequence, they would struggle to deal with the semantic variations across domains and to build a category-discriminative embedding space, leading to less discrimination and poor generalization. In this work, we propose Semantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixel to promote learning of class-discriminative and class-balanced pixel embedding space across domains, eventually boosting the performance of self-training methods. Specifically, to explore proper semantic concepts, we first investigate a centroid-aware pixel contrast that employs the category centroids of the entire source domain or a single source image to guide the learning of discriminative features. Considering the possible lack of category diversity in semantic concepts, we then blaze a trail of distributional perspective to involve a sufficient quantity of instances, namely distribution-aware pixel contrast, in which we approximate the true distribution of each semantic category from the statistics of labeled source data. Moreover, such an optimization objective can derive a closed-form upper bound by implicitly involving an infinite number of (dis)similar pairs, making it computationally efficient. Extensive experiments show that SePiCo not only helps stabilize training but also yields discriminative features, making significant progress on both synthetic-to-real and daytime-to-nighttime scenarios. Most notably, SePiCo establishes state-of-the-art performance on tasks of GTAV → Cityscapes, SYNTHIA → Cityscapes and Cityscapes → Dark Zurich, improving by 12.8% mIoU, 8.8% mIoU, and 9.2% mIoU compared to the previous best method, respectively. The code will be available at https://github.com/BIT-DA/SePiCo.

show abstract

“…In other words, previous works for MD do not take into account inputs lying outside of the training data distribution. Therefore, a single dataset such as CIFAR-10/100 [19] is sufficient to evaluate the MD performance of a model [20,21]. OsR and OoDD can be categorized as the same problem; however, the benchmark settings are slightly different in the literature.…”

Section: Introductionmentioning

confidence: 99%

A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks

Kim¹,

Koo²,

Hwang³

2021

Preprint

View full text Add to dashboard Cite

Deep neural networks have achieved outstanding performance over various tasks, but they have a critical issue: over-confident predictions even for completely unknown samples. Many studies have been proposed to successfully filter out these unknown samples, but they only considered narrow and specific tasks, referred to as misclassification detection, open-set recognition, or outof-distribution detection. In this work, we argue that these tasks should be treated as fundamentally an identical problem because an ideal model should possess detection capability for all those tasks. Therefore, we introduce the unknown detection task, an integration of previous individual tasks, for a rigorous examination of the detection capability of deep neural networks on a wide spectrum of unknown samples. To this end, unified benchmark datasets on different scales were constructed and the unknown detection capabilities of existing popular methods were subject to comparison. We found that Deep Ensemble consistently outperforms the other approaches in detecting unknowns; however, all methods are only successful for a specific type of unknown. The reproducible code and benchmark datasets are available at https://github.com/daintlab/unknown-detection-benchmarks.

show abstract

Confidence Estimation via Auxiliary Models

Cited by 39 publications

References 35 publications

Judgment aggregation, discursive dilemma and reflective equilibrium: Neural language models as self-improving doxastic agents

Judgment aggregation, discursive dilemma and reflective equilibrium: Neural language models as self-improving doxastic agents

SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation

A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks

Contact Info

Product

Resources

About