Saliency Guided Self-Attention Network for Weakly and Semi-Supervised Semantic Segmentation

Abstract

Weakly supervised semantic segmentation (WSSS) using only image-level labels can greatly reduce the annotation cost and therefore has attracted considerable research interest. However, its performance is still inferior to the fully supervised counterparts. To mitigate the performance gap, we propose a saliency guided self-attention network (SGAN) to address the WSSS problem. The introduced self-attention mechanism is able to capture rich and extensive contextual information but also may mis-spread attentions to unexpected regions. To enable this mechanism work effectively under weak supervision, we integrate class-agnostic saliency priors into the self-attention mechanism to prevent the attentions on discriminative parts from misspreading to the background. And meanwhile we utilize classspecific attention cues as an additional supervision for SGAN, which reduces the mis-spread of attentions in regions belonging to different foreground categories. The proposed approach is able to produce dense and accurate localization cues, by which the segmentation performance is boosted. Experiments on PASCAL VOC 2012 dataset show that the proposed approach outperforms all other state-of-the-art methods.Abstract-Weakly supervised semantic segmentation (WSSS) using only image-level labels can greatly reduce the annotation cost and therefore has attracted considerable research interest. However, its performance is still inferior to the fully supervised counterparts. To mitigate the performance gap, we propose a saliency guided self-attention network (SGAN) to address the WSSS problem. The introduced self-attention mechanism is able to capture rich and extensive contextual information but also may mis-spread attentions to unexpected regions. To enable this mechanism work effectively under weak supervision, we integrate class-agnostic saliency priors into the self-attention mechanism to prevent the attentions on discriminative parts from misspreading to the background. And meanwhile we utilize classspecific attention cues as an additional supervision for SGAN, which reduces the mis-spread of attentions in regions belonging to different foreground categories. The proposed approach is able to produce dense and accurate localization cues, by which the segmentation performance is boosted. Experiments on PASCAL VOC 2012 dataset show that the proposed approach outperforms all other state-of-the-art methods.

Extracting Structured Supervision From Captions for Weakly Supervised Semantic Segmentation

“…In particular, we observe that even our VGG-16-based model outperforms all previous methods trained using only ground truth classification labels, showing the effectiveness of our approach, and of image captions as supervision for this task. Our method also presents comparable results to those reported in [43] for SGAN, despite the fact that this model leverages much stronger supervision extensively in the form of class-agnostic saliency maps, generated by a model trained with pixel-level supervision.…”

“…In this work, we focus on a different but complemen- [21], [25], [27], [38] labels - [24], [26], [39]- [43] labels tary problem: generating accurate segmentation masks using natural language captions instead of classification labels, which is not possible using these previous methods. Our method also introduces an effective way to take advantage of complementary labels for background categories and visual attributes for mask generation, which cannot be leveraged with existing systems.…”

“…Our method also introduces an effective way to take advantage of complementary labels for background categories and visual attributes for mask generation, which cannot be leveraged with existing systems. Some of these previous studies also used class-agnostic saliency maps to generate background cues [22]- [24], [26], [39], while others used saliency maps to guide and refine foreground regions [40]- [43]. However, this introduces dependencies on off-the-shelf models that require stronger supervision.…”

“…Most of the reported experiments were performed using the MS-COCO dataset [30], which contains 123k images, each annotated with 5 different textual captions, as well as instance-level segmentation masks for 80 object categories. Following previous studies [23], [27], [33], [43], we used the train2014 split with 83k images for training, and reserved the val2014 split with 41k images exclusively for testing. To adjust the hyperparameters of our model, we followed the same validation strategy as in [33], which consisted in evaluating the quality of the generated masks on the training set.…”

“…As in previous studies [23], [27], [33], [43], performance is reported in terms of pixel-wise mean Intersection-over-Union (mIoU), averaged across all categories.…”