3D Guided Weakly Supervised Semantic Segmentation

Sun, Weixuan; Zhang, Jing; Barnes, Nick

doi:10.1007/978-3-030-69525-5_35

Cited by 11 publications

(4 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To save labeling cost, many WSSS methods have been proposed, including those using image-level labels [2,8,37,47,65,70,72], scribbles [39], points [6], and bounding boxes [16,36,46,56]. We mainly focus on image-level models, which can be grouped into two families: multi-step, and one-step end-to-end methods.…”

Section: Weakly Supervised Semantic Segmentationmentioning

confidence: 99%

GETAM: Gradient-weighted Element-wise Transformer Attention Map for Weakly-supervised Semantic segmentation

Sun¹,

Zhang²,

Liu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Weakly-supervised semantic segmentation (WSSS) is challenging, particularly when image-level labels are used to supervise pixel-level prediction. To bridge their gap, a Class Activation Map (CAM) is usually generated to provide pixel-level pseudo labels. CAMs in Convolutional Neural Networks suffer from partial activation i.e. only the most discriminative regions are activated. Transformerbased methods, on the other hand, are highly effective at exploring global context, potentially alleviating the "partial activation" issue. In this paper, we propose the first transformer-based WSSS approach, and introduce the Gradient-weighted Element-wise Transformer Attention Map (GETAM). GETAM shows fine scale activation for all feature map elements, revealing different parts of the object across transformer layers. Further, we propose an activation-aware label-completion module to generate high-quality pseudo labels. Finally, we incorporate our methods into an end-to-end framework for WSSS using double-backward propagation. Extensive experiments on PASCAL VOC and COCO dataset demonstrate that our results outperform not only the state-of-the-art end-to-end approaches by a significant margin, but also most of the multi-stage methods.

show abstract

Section: Weakly Supervised Semantic Segmentationmentioning

confidence: 99%

GETAM: Gradient-weighted Element-wise Transformer Attention Map for Weakly-supervised Semantic segmentation

Sun¹,

Zhang²,

Liu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Weakly Supervised Semantic Segmentation: A large number of WSSS methods have been proposed to achieve a trade-off between labeling efficiency and model accuracy, where the "weak" annotations can be image-level labels [23,44,2,16,34,42,5,50,6,37,48,18,47,13,28,46,45,29], scribbles [40,31,39], points [3], or bounding boxes [10,34,25,30,36,38]. We mainly focus on image-level label based weakly supervised models.…”

Section: Related Workmentioning

confidence: 99%

Inferring the Class Conditional Response Map for Weakly Supervised Semantic Segmentation

Sun¹,

Zhang²,

Barnes³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Image-level weakly supervised semantic segmentation (WSSS) relies on class activation maps (CAMs) for pseudo labels generation. As CAMs only highlight the most discriminative regions of objects, the generated pseudo labels are usually unsatisfactory to serve directly as supervision. To solve this, most existing approaches follow a multitraining pipeline to refine CAMs for better pseudo-labels, which includes: 1) re-training the classification model to generate CAMs; 2) post-processing CAMs to obtain pseudo labels; and 3) training a semantic segmentation model with the obtained pseudo labels. However, this multi-training pipeline requires complicated adjustment and additional time. To address this, we propose a class-conditional inference strategy and an activation aware mask refinement loss function to generate better pseudo labels without retraining the classifier. The class conditional inference-time approach is presented to separately and iteratively reveal the classification network's hidden object activation to generate more complete response maps. Further, our activation aware mask refinement loss function introduces a novel way to exploit saliency maps during segmentation training and refine the foreground object masks without suppressing background objects. Our method achieves superior WSSS results without requiring re-training of the classifier. https://github.com/weixuansun/InferCam

show abstract

“…Porzi et al [43] refine a segmentation model by using tracking and optical flow to improve predictions into automatic annotations. Sun et al [56] map RGB-D images into a point cloud, autonomously labeling these point clouds in 3D space by leveraging bounding box annotations and projecting them back onto the image, in order to obtain pseudo labels that can be used for training.…”

Section: B Self-supervision and Pseudolabelsmentioning

confidence: 99%

Self-Improving Semantic Perception for Indoor Localisation

Blum¹,

Milano²,

Zurbrügg³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a novel robotic system that can improve its semantic perception during deployment. Contrary to the established approach of learning semantics from large datasets and deploying fixed models, we propose a framework in which semantic models are continuously updated on the robot to adapt to the deployment environments. Our system therefore tightly couples multi-sensor perception and localisation to continuously learn from self-supervised pseudo labels. We study this system in the context of a construction robot registering LiDAR scans of cluttered environments against building models. Our experiments show how the robot's semantic perception improves during deployment and how this translates into improved 3D localisation by filtering the clutter out of the LiDAR scan, even across drastically different environments. We further study the risk of catastrophic forgetting that such a continuous learning setting poses. We find memory replay an effective measure to reduce forgetting and show how the robotic system can improve even when switching between different environments. On average, our system improves by 60% in segmentation and 10% in localisation compared to deployment of a fixed model, and it keeps this improvement up while adapting to further environments.

show abstract

3D Guided Weakly Supervised Semantic Segmentation

Cited by 11 publications

References 45 publications

GETAM: Gradient-weighted Element-wise Transformer Attention Map for Weakly-supervised Semantic segmentation

GETAM: Gradient-weighted Element-wise Transformer Attention Map for Weakly-supervised Semantic segmentation

Inferring the Class Conditional Response Map for Weakly Supervised Semantic Segmentation

Self-Improving Semantic Perception for Indoor Localisation

Contact Info

Product

Resources

About