Detecting the Unexpected via Image Resynthesis

Lis, Krzysztof; Nakka, Krishna Kanth; Fua, Pascal; Salzmann, Mathieu

doi:10.1109/iccv.2019.00224

Cited by 147 publications

(156 citation statements)

References 29 publications

Supporting

Mentioning

154

Contrasting

Order By: Relevance

“…2 scale poorly to the level of detail in urban driving, good results have been achieved with generative adversarial networks (Wang et al, 2018;Isola et al, 2017) that synthesize driving scenes from semantic segmentation. Lis et al (2019) uses such a method to find outliers by comparing the original and resynthesized image, where they train the comparison on flipped semantic labels in the ID data and therefore do not require outliers in training. While the original work (Lis et al, 2019) experimented with lower resolution segmentation data, Di Biase et al (2021) submitted an adapted, scaled-up model.…”

Section: Submitted Methodsmentioning

confidence: 99%

“…Lis et al (2019) uses such a method to find outliers by comparing the original and resynthesized image, where they train the comparison on flipped semantic labels in the ID data and therefore do not require outliers in training. While the original work (Lis et al, 2019) experimented with lower resolution segmentation data, Di Biase et al (2021) submitted an adapted, scaled-up model.…”

Section: Submitted Methodsmentioning

confidence: 99%

See 1 more Smart Citation

The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

et al. 2021

View full text Add to dashboard Cite

Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for anomaly detection in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects. We adapt state-of-the-art methods to recent semantic segmentation models and compare uncertainty estimation approaches based on softmax confidence, Bayesian learning, density estimation, image resynthesis, as well as supervised anomaly detection methods. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art. Results, data and submission information can be found at https://fishyscapes.com/.

show abstract

Section: Submitted Methodsmentioning

confidence: 99%

Section: Submitted Methodsmentioning

confidence: 99%

The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Instead of comparing each pixel of the input image with the one in the resynthesized image directly, Lis et al (2019) propose to train a discrepancy network on artificially generated anomalies that directly outputs the regions where the reconstruction failed. Since their method requires pixelprecise semantic annotations of the training data, we do not consider this method for our benchmark.…”

Section: Generative Adversarial Network (Gans)mentioning

confidence: 99%

The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection

et al. 2021

View full text Add to dashboard Cite

The detection of anomalous structures in natural image data is of utmost importance for numerous tasks in the field of computer vision. The development of methods for unsupervised anomaly detection requires data on which to train and evaluate new approaches and ideas. We introduce the MVTec anomaly detection dataset containing 5354 high-resolution color images of different object and texture categories. It contains normal, i.e., defect-free images intended for training and images with anomalies intended for testing. The anomalies manifest themselves in the form of over 70 different types of defects such as scratches, dents, contaminations, and various structural changes. In addition, we provide pixel-precise ground truth annotations for all anomalies. We conduct a thorough evaluation of current state-of-the-art unsupervised anomaly detection methods based on deep architectures such as convolutional autoencoders, generative adversarial networks, and feature descriptors using pretrained convolutional neural networks, as well as classical computer vision methods. We highlight the advantages and disadvantages of multiple performance metrics as well as threshold estimation techniques. This benchmark indicates that methods that leverage descriptors of pretrained networks outperform all other approaches and deep-learning-based generative models show considerable room for improvement.

show abstract

“…Anomaly detection by reconstruction. Anomalies can be detected by training an autoencoder [2], [11] or generative model [39], [57] on in-distribution data and use the quality of the reconstruction as a proxy OOD, as the autoencoder is unlikely to decode accurately patterns not seen during training.…”

Section: Related Workmentioning

confidence: 99%

One Versus All for Deep Neural Network for Uncertainty (OVNNI) Quantification

Franchi¹,

Bursuc²,

Aldea³

et al. 2022

IEEE Access

View full text Add to dashboard Cite

Deep neural networks (DNNs) are powerful learning models yet their results are not always reliable. This drawback results from the fact that modern DNNs are usually overconfident, and consequently their epistemic uncertainty cannot be straightforwardly characterized. In this work, we propose a new technique to quantify the epistemic uncertainty of data easily. This method consists in mixing the predictions of an ensemble of DNNs trained to classify One class vs. All the other classes (OVA) with predictions from a standard DNN trained to perform All vs. All (AVA) classification. First of all, the adjustment provided by the AVA DNN to the score of the base classifiers allows for a more fine-grained inter-class separation. Moreover, the two types of classifiers enforce mutually their detection of out-of-distribution (OOD) samples, circumventing entirely the requirement of using such samples during training. The additional cost involved by the construction of the ensemble is offset by the ease of use of our proposed strategy and by its enhanced generalization potential, as it does not bind its performance in a given context to specific OOD datasets. The extensive experiments confirm the wide applicability of our approach, and our method achieves state of the art performance in quantifying OOD data across multiple datasets and architectures while requiring little hyper-parameter tuning.INDEX TERMS Uncertainty estimation, DNN ensembles, one vs all classification, all vs all classification.

show abstract

Detecting the Unexpected via Image Resynthesis

Cited by 147 publications

References 29 publications

The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection

One Versus All for Deep Neural Network for Uncertainty (OVNNI) Quantification

Contact Info

Product

Resources

About