Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers

Ru, Lixiang; Zhan, Yibing; Yu, Baosheng; Du, Bo

doi:10.1109/cvpr52688.2022.01634

Cited by 159 publications

(101 citation statements)

References 37 publications

Supporting

Mentioning

100

Contrasting

Order By: Relevance

“…Observe that we improved BPM state-of-the-art by 3.91 mIou% points and BPM+CRF by 5.4 mIoU%, both w.r.t. AFA [39], owning so far the best accuracy on both. B.…”

Section: Comparisons With State-of-the-artmentioning

confidence: 99%

“…This last is connected with the class prediction, using a BCE prediction loss. More recently, Vision Transformers [15] are emerging as an alternative to generate CAM [58,39]. Our method is the first one using only ViT without CAM to generate baseline pseudo-masks.…”

Section: Related Workmentioning

confidence: 99%

“…Dino [5] downstream task segments foreground from background for single class images, differing from WSSS. Only recently ViT contributed to WSSS with MCTformer [58] and AFA [39], though both resort to CAM. MCTformer exploits ViT attention mechanism to obtain localization maps.…”

Section: Related Workmentioning

confidence: 99%

“…Our ViT-PCM, with backbone ViT-S/16, is green-dashed, and ViT-B/16 is green-continuous. We can observe that most of the shown methods are multi-stage (see also [3,39]), and boosting the BPM asks for a significant increase of parameters. Table 3 shows the accuracy between CAM and PCM on different backbones and the amount of parameters required.…”

Section: Comparisons With State-of-the-artmentioning

confidence: 99%

“…[6] and [41] use PSA; CONTA [14] and ReCAM [12] use IRNet while [30] using both. AFA [39] use PAC [44].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation

Rossetti¹,

Zappia²,

Sanzari³

et al. 2022

Preprint

View full text Add to dashboard Cite

Weakly Supervised Semantic Segmentation (WSSS) research has explored many directions to improve the typical pipeline CNN plus class activation maps (CAM) plus refinements, given the image-class label as the only supervision. Though the gap with the fully supervised methods is reduced, further abating the spread seems unlikely within this framework. On the other hand, WSSS methods based on Vision Transformers (ViT) have not yet explored valid alternatives to CAM. ViT features have been shown to retain a scene layout, and object boundaries in self-supervised learning. To confirm these findings, we prove that the advantages of transformers in self-supervised methods are further strengthened by Global Max Pooling (GMP), which can leverage patch features to negotiate pixel-label probability with class probability. This work proposes a new WSSS method dubbed ViT-PCM (ViT Patch-Class Mapping), not based on CAM. The end-to-end presented network learns with a single optimization process, refined shape and proper localization for segmentation masks. Our model outperforms the state-of-the-art on baseline pseudo-masks (BPM), where we achieve 69.3% mIoU on Pas-calVOC 2012 val set. We show that our approach has the least set of parameters, though obtaining higher accuracy than all other approaches. In a sentence, quantitative and qualitative results of our method reveal that ViT-PCM is an excellent alternative to CNN-CAM based architectures.

show abstract

“…Observe that we improved BPM state-of-the-art by 3.91 mIou% points and BPM+CRF by 5.4 mIoU%, both w.r.t. AFA [39], owning so far the best accuracy on both. B.…”

Section: Comparisons With State-of-the-artmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Comparisons With State-of-the-artmentioning

confidence: 99%

“…[6] and [41] use PSA; CONTA [14] and ReCAM [12] use IRNet while [30] using both. AFA [39] use PAC [44].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation

Rossetti¹,

Zappia²,

Sanzari³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Dual‐branch Transformer for semi‐supervised medical image segmentation

Huang,

Zhu,

Shao

et al. 2024

J Applied Clin Med Phys

View full text Add to dashboard Cite

PurposeIn recent years, the use of deep learning for medical image segmentation has become a popular trend, but its development also faces some challenges. Firstly, due to the specialized nature of medical data, precise annotation is time‐consuming and labor‐intensive. Training neural networks effectively with limited labeled data is a significant challenge in medical image analysis. Secondly, convolutional neural networks commonly used for medical image segmentation research often focus on local features in images. However, the recognition of complex anatomical structures or irregular lesions often requires the assistance of both local and global information, which has led to a bottleneck in its development. Addressing these two issues, in this paper, we propose a novel network architecture.MethodsWe integrate a shift window mechanism to learn more comprehensive semantic information and employ a semi‐supervised learning strategy by incorporating a flexible amount of unlabeled data. Specifically, a typical U‐shaped encoder‐decoder structure is applied to obtain rich feature maps. Each encoder is designed as a dual‐branch structure, containing Swin modules equipped with windows of different size to capture features of multiple scales. To effectively utilize unlabeled data, a level set function is introduced to establish consistency between the function regression and pixel classification.ResultsWe conducted experiments on the COVID‐19 CT dataset and DRIVE dataset and compared our approach with various semi‐supervised and fully supervised learning models. On the COVID‐19 CT dataset, we achieved a segmentation accuracy of up to 74.56%. Our segmentation accuracy on the DRIVE dataset was 79.79%.ConclusionsThe results demonstrate the outstanding performance of our method on several commonly used evaluation metrics. The high segmentation accuracy of our model demonstrates that utilizing Swin modules with different window sizes can enhance the feature extraction capability of the model, and the level set function can enable semi‐supervised models to more effectively utilize unlabeled data. This provides meaningful insights for the application of deep learning in medical image segmentation. Our code will be released once the manuscript is accepted for publication.

show abstract

Max Pooling with Vision Transformers Reconciles Class and Shape in Weakly Supervised Semantic Segmentation

Rossetti

Zappia²,

Sanzari

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers

Cited by 159 publications

References 37 publications

Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation

Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation

Dual‐branch Transformer for semi‐supervised medical image segmentation

Max Pooling with Vision Transformers Reconciles Class and Shape in Weakly Supervised Semantic Segmentation

Contact Info

Product

Resources

About