GETAM: Gradient-weighted Element-wise Transformer Attention Map for Weakly-supervised Semantic segmentation

Abstract

Weakly-supervised semantic segmentation (WSSS) is challenging, particularly when image-level labels are used to supervise pixel-level prediction. To bridge their gap, a Class Activation Map (CAM) is usually generated to provide pixel-level pseudo labels. CAMs in Convolutional Neural Networks suffer from partial activation i.e. only the most discriminative regions are activated. Transformerbased methods, on the other hand, are highly effective at exploring global context, potentially alleviating the "partial activation" issue. In this paper, we propose the first transformer-based WSSS approach, and introduce the Gradient-weighted Element-wise Transformer Attention Map (GETAM). GETAM shows fine scale activation for all feature map elements, revealing different parts of the object across transformer layers. Further, we propose an activation-aware label-completion module to generate high-quality pseudo labels. Finally, we incorporate our methods into an end-to-end framework for WSSS using double-backward propagation. Extensive experiments on PASCAL VOC and COCO dataset demonstrate that our results outperform not only the state-of-the-art end-to-end approaches by a significant margin, but also most of the multi-stage methods.

Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learning

“…Inspired by the popular Grad-CAM technique (Selvaraju et al, 2017), we incorporated head-wise gradient-weighing for self-attention maps to boost the presentation of the weights relevant to specific class activation for the first time. Compared with other attention mapping techniques (Chefer et al, 2021;Sun et al, 2021) that relied on the ViT, we were also the first to implement it on the more complex Swin transformer that was intended to improve upon the ViT. The enhanced visualization of the attention maps and ICH segmentation accuracy are evident in Fig.…”

“…For instance, Chefer et al (2021) utilize the Taylor Decomposition principle to assign and propagate a local relevance score through the layers of a ViT model. Similarly, Sun et al (2021) and Barkan et al (2021) employ attention gradient weighting on ViT and BERT models, respectively. However, these approaches primarily focused on the attention weight of the "cls" token, and the latter two methods weighed each token's attention weight through element-wise multiplication.…”

Weakly Supervised Intracranial Hemorrhage Segmentation Using Hierarchical Combination of Attention Maps from a Swin Transformer

“…Inspired by the popular Grad-CAM technique (Selvaraju et al, 2017), we incorporated head-wise gradient-weighing for self-attention maps to boost the presentation of the weights relevant to specific class activation for the first time. Compared with other attention mapping technique (Chefer et al, 2021;Sun et al, 2021) that relied on the ViT, we were also the first to implement it on the more complex Swin transformer that was intended to improve upon the ViT. The enhanced visualization of the attention maps and ICH segmentation accuracy are evident in Fig.…”