The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

Blum, Hermann; Sarlin, Paul-Edouard; Nieto, Juan; Siegwart, Roland; Cadena, César

doi:10.1007/s11263-021-01511-6

Cited by 72 publications

(66 citation statements)

References 39 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The resulting open-set classifiers produce predictions over a set of K known classes and one unknown class (section 2.2). Things get more complicated in the case of dense prediction where we have to deal with outlier objects in inlier scenes [15,12,8] (section 2.3).…”

Section: Related Workmentioning

confidence: 99%

“…Previous approaches to dense anomaly detection rely on image resynthesis [7,8,9], Bayesian modeling [10,11], recognition in the latent space [12,13], or auxiliary negative training data [14,15]. However, all these approaches have significant shortcomings.…”

Section: Introductionmentioning

confidence: 99%

“…In any case, these methods are not suitable for real-time applications. Recognition in latent space [12] may be sensitive to feature collapse [16,17] due to relying on a pretrained classifier. Reliance on auxiliary negative data may introduce several kinds of undesired bias such as overoptimistic performance on some test images.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dense anomaly detection by robust learning on synthetic negative data

Grcić¹,

Bevandić²,

Šegvić³

2021

Preprint

View full text Add to dashboard Cite

Standard machine learning is unable to accommodate inputs which do not belong to the training distribution. The resulting models often give rise to confident incorrect predictions which may lead to devastating consequences. This problem is especially demanding in the context of dense prediction since input images may be only partially anomalous. Previous work has addressed dense anomaly detection by discriminative training on mixed-content images. We extend this approach with synthetic negative patches which simultaneously achieve high inlier likelihood and uniform discriminative prediction. We generate synthetic negatives with normalizing flows due to their outstanding distribution coverage and capability to generate samples at different resolutions. We also propose to detect anomalies according to a principled information-theoretic criterion which can be consistently applied through training and inference. The resulting models set the new state of the art on standard benchmarks and datasets in spite of minimal computational overhead and refraining from auxiliary negative data.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dense anomaly detection by robust learning on synthetic negative data

Grcić¹,

Bevandić²,

Šegvić³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…It has been noted that out-of-distribution images can be identified through epistemic uncertainties [15]. Many other per-pixel uncertainty measures have also been developed and compared in [1] including ODIN [19], Bayesian networks [14,23], density estimation [4], and OOD training [3].…”

Section: Related Work Fusion Architectures For Rgb-d Semantic Segment...mentioning

confidence: 99%

“…The uncertainty metrics provide pixel-wise scores, which we average over an entire image. We perform this averaging step because the per-pixel uncertainty metrics can be unreliable as a local indicator of deviation, as shown by [1]. Thus, from the three uncertainty metrics listed above, three deviation ratios can be calculated: In the end, we combine the deviation ratios of different metrics using a Min operation.…”

Section: A Uncertainty Estimation and Deviation Ratiomentioning

confidence: 99%

UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

Tian

Cheung

Glaser

et al. 2019

Preprint

View full text Add to dashboard Cite

The fusion of multiple sensor modalities, especially through deep learning architectures, has been an active area of study. However, an under-explored aspect of such work is whether the methods can be robust to degradations across their input modalities, especially when they must generalize to degradations not seen during training. In this work, we propose an uncertainty-aware fusion scheme to effectively fuse inputs that might suffer from a range of known and unknown degradations. Specifically, we analyze a number of uncertainty measures, each of which captures a different aspect of uncertainty, and we propose a novel way to fuse degraded inputs by scaling modalityspecific output softmax probabilities. We additionally propose a novel data-dependent spatial temperature scaling method to complement these existing uncertainty measures. Finally, we integrate the uncertainty-scaled output from each modality using a probabilistic noisy-or fusion method. In a photo-realistic simulation environment (AirSim), we show that our method achieves significantly better results on a semantic segmentation task, compared to state-of-art fusion architectures, on a range of degradations (e.g. fog, snow, frost, and various other types of noise), some of which are unknown during training. We specifically improve upon the state-of-art [30] by 28% in mean IoU on various degradations.

show abstract

Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift

Bevandić

Krešo

Oršić

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Recent success on realistic road driving datasets has increased interest in exploring robust performance in real-world applications. One of the major unsolved problems is to identify image content which can not be reliably recognized with a given inference engine. We therefore study approaches to recover a dense outlier map alongside the primary task with a single forward pass, by relying on shared convolutional features. We consider semantic segmentation as the primary task and perform extensive validation on WildDash val (inliers), LSUN val (outliers), and pasted objects from Pascal VOC 2007 (outliers). We achieve the best validation performance by training to discriminate inliers from pasted ImageNet-1k content, even though ImageNet-1k contains many roaddriving pixels, and, at least nominally, fails to account for the full diversity of the visual world. The proposed two-head model performs comparably to the C-way multi-class model trained to predict uniform distribution in outliers, while outperforming several other validated approaches. We evaluate our best two models on the WildDash test dataset and set a new state of the art on the WildDash benchmark.

show abstract

The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

Cited by 72 publications

References 39 publications

Dense anomaly detection by robust learning on synthetic negative data

Dense anomaly detection by robust learning on synthetic negative data

UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift

Contact Info

Product

Resources

About