Bringing Background into the Foreground: Making All Classes Equal in Weakly-Supervised Video Semantic Segmentation

Saleh, Fatemeh Sadat; Aliakbarian, Mohammad Sadegh; Salzmann, Mathieu; Álvarez, Jose M.

doi:10.1109/iccv.2017.232

Cited by 35 publications

(23 citation statements)

References 66 publications

(154 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results on CamVid in Table 4, where we compare our method to fullysupervised techniques that make use of CamVid images and annotations to train a model, GTA5-based baselines, and the state-of-the-art weakly-supervised method, show a similar trend. Our approach clearly outperforms the weaklysupervised method of [39] and a DeepLab semantic segmentation network trained on synthetic data. In fact, on this dataset, it event outperforms some of the fully supervised methods that rely on annotated CamVid images for training.…”

Section: Resultsmentioning

confidence: 90%

“…In Table 3, we compare our approach with the state-of-the-are weakly-supervised method of [39] and with state-of-the-art domain adaptation methods. The results for these methods were directly taken from their respective papers.…”

Section: Resultsmentioning

confidence: 99%

“…However, having detailed information about the different background classes is crucial in many practical scenarios, such as automated driving, where one needs to differentiate, e.g., the road from a grass field. To the best of our knowledge, [39] constitutes the only method that considers multiple background classes for weakly-supervised semantic segmentation. This is achieved by leveraging both appearance and motion via a two-stream architecture trained using a loss based on classifier heatmaps.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Effective Use of Synthetic Data for Urban Scene Semantic Segmentation

Saleh

Aliakbarian

Salzmann

et al. 2018

Lecture Notes in Computer Science

Self Cite

127

View full text Add to dashboard Cite

Training a deep network to perform semantic segmentation requires large amounts of labeled data. To alleviate the manual effort of annotating real images, researchers have investigated the use of synthetic data, which can be labeled automatically. Unfortunately, a network trained on synthetic data performs relatively poorly on real images. While this can be addressed by domain adaptation, existing methods all require having access to real images during training. In this paper, we introduce a drastically different way to handle synthetic images that does not require seeing any real images at training time. Our approach builds on the observation that foreground and background classes are not affected in the same manner by the domain shift, and thus should be treated differently. In particular, the former should be handled in a detection-based manner to better account for the fact that, while their texture in synthetic images is not photo-realistic, their shape looks natural. Our experiments evidence the effectiveness of our approach on Cityscapes and CamVid with models trained on synthetic data only.

show abstract

Section: Resultsmentioning

confidence: 90%

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Effective Use of Synthetic Data for Urban Scene Semantic Segmentation

Saleh

Aliakbarian

Salzmann

et al. 2018

Lecture Notes in Computer Science

Self Cite

127

View full text Add to dashboard Cite

show abstract

“…Furthermore, most existing works are dedicated to handle multiple salient foreground instances and evaluate on the Pascal VOC dataset [12]. [36] is the only existing work that considers complete scene parsing (background + foreground) with only imagelevel label by leveraging two-stream deep architecture and heat map loss. However, their result does not perform well compared to other adaptation methods on the Cityscape dataset.…”

Section: Weak Image-level Supervisionmentioning

confidence: 99%

What Synthesis Is Missing: Depth Adaptation Integrated With Weak Supervision for Indoor Scene Parsing

Liu

Shen

Klopp

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Scene Parsing is a crucial step to enable autonomous systems to understand and interact with their surroundings. Supervised deep learning methods have made great progress in solving scene parsing problems, however, come at the cost of laborious manual pixel-level annotation. To alleviate this effort synthetic data as well as weak supervision have both been investigated. Nonetheless, synthetically generated data still suffers from severe domain shift while weak labels are often imprecise. Moreover, most existing works for weakly supervised scene parsing are limited to salient foreground objects. The aim of this work is hence twofold: Exploit synthetic data where feasible and integrate weak supervision where necessary. More concretely, we address this goal by utilizing depth as transfer domain because its synthetic-to-real discrepancy is much lower than for color. At the same time, we perform weak localization from easily obtainable image level labels and integrate both using a novel contour-based scheme. Our approach is implemented as a teacher-student learning framework to solve the transfer learning problem by generating a pseudo ground truth. Using only depth-based adaptation, this approach already outperforms previous transfer learning approaches on the popular indoor scene parsing SUN RGB-D dataset. Our proposed two-stage integration more than halves the gap towards fully supervised methods when compared to previous state-of-the-art in transfer learning. Weak Localizations …… … Pseudo Labels Source Domain Synthetic Data Depth Target Domain Real Data Depth RGB Domain Adaptation Teacher …… Student Fusion Depth-based Predictions Teacher arXiv:1903.09781v1 [cs.CV]

show abstract

“…Recently, weakly-supervised methods for video object segmentation [40,42,31,41] have been developed to relax the need for annotations where only class-level labels are required. These approaches have significantly reduced the labor-intensive step of collecting pixel-wise training data on target categories.…”

Section: Introductionmentioning

confidence: 99%

Unseen Object Segmentation in Videos via Transferable Representations

Chen

Tsai²,

Yang

et al. 2019

Computer Vision – ACCV 2018

View full text Add to dashboard Cite

In order to learn object segmentation models in videos, conventional methods require a large amount of pixel-wise ground truth annotations. However, collecting such supervised data is time-consuming and labor-intensive. In this paper, we exploit existing annotations in source images and transfer such visual information to segment videos with unseen object categories. Without using any annotations in the target video, we propose a method to jointly mine useful segments and learn feature representations that better adapt to the target frames. The entire process is decomposed into two tasks: 1) solving a submodular function for selecting object-like segments, and 2) learning a CNN model with a transferable module for adapting seen categories in the source domain to the unseen target video. We present an iterative update scheme between two tasks to self-learn the final solution for object segmentation. Experimental results on numerous benchmark datasets show that the proposed method performs favorably against the state-of-the-art algorithms.

show abstract

Bringing Background into the Foreground: Making All Classes Equal in Weakly-Supervised Video Semantic Segmentation

Cited by 35 publications

References 66 publications

Effective Use of Synthetic Data for Urban Scene Semantic Segmentation

Effective Use of Synthetic Data for Urban Scene Semantic Segmentation

What Synthesis Is Missing: Depth Adaptation Integrated With Weak Supervision for Indoor Scene Parsing

Unseen Object Segmentation in Videos via Transferable Representations

Contact Info

Product

Resources

About