Multi-class Token Transformer for Weakly Supervised Semantic Segmentation

Abstract

This paper proposes a new transformer-based framework to learn class-specific object localization maps as pseudo labels for weakly supervised semantic segmentation (WSSS). Inspired by the fact that the attended regions of the one-class token in the standard vision transformer can be leveraged to form a class-agnostic localization map, we investigate if the transformer model can also effectively capture class-specific attention for more discriminative object localization by learning multiple class tokens within the transformer. To this end, we propose a Multi-class Token Transformer, termed as MCTformer, which uses multiple class tokens to learn interactions between the class tokens and the patch tokens. The proposed MCTformer can successfully produce class-discriminative object localization maps from the class-to-patch attentions corresponding to different class tokens. We also propose to use a patchlevel pairwise affinity, which is extracted from the patchto-patch transformer attention, to further refine the localization maps. Moreover, the proposed framework is shown to fully complement the Class Activation Mapping (CAM) method, leading to remarkably superior WSSS results on the PASCAL VOC and MS COCO datasets. These results underline the importance of the class token for WSSS. 1

1st Place Solution for ECCV 2022 OOD-CV Challenge Object Detection Track

“…For data augmentation, we apply mixup [17], color jitter, foggy [6], snow [6], resize, horizontal flipping, rotation, blur, noise, random erase [19], and mask-level copy-paste. To obtain more vivid occlusion effects, we introduce MCTformer [15] to generate pseudo masks of the salient objects in the subset of ImageNet-1K. Then, these masks are used in the mask-level copy-paste strategy [5], which are pasted onto the training images to simulate object occlusion.…”

Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation

“…This last is connected with the class prediction, using a BCE prediction loss. More recently, Vision Transformers [15] are emerging as an alternative to generate CAM [58,39]. Our method is the first one using only ViT without CAM to generate baseline pseudo-masks.…”

“…Dino [5] downstream task segments foreground from background for single class images, differing from WSSS. Only recently ViT contributed to WSSS with MCTformer [58] and AFA [39], though both resort to CAM. MCTformer exploits ViT attention mechanism to obtain localization maps.…”

“…Our ViT-PCM has the best accuracy among the end-to-end methods, with 70.3% and 70.9% on val and test sets. Our method is second to MCT-Former [58] on the test set w.r.t. all methods (boosted and end-to-end).…”

MVS-T: A Coarse-to-Fine Multi-View Stereo Network with Transformer for Low-Resolution Images 3D Reconstruction

“…Recently, neural network design in natural language processing (NLP) has embarked on a completely different path since Transformer [ 18 ] has replaced recurrent neural networks as the dominant network architecture. With the introduction of Vision Transformers (ViT) [ 19 ], more and more scholars apply Transformer to computer vision [ 20 , 21 , 22 , 23 , 24 , 25 ]. The Transformer’s superior design architecture and self-attention mechanism can better model spatial relationships and aggregate features at arbitrary locations.…”