Cap2Seg: Inferring Semantic and Spatial Context from Captions for Zero-Shot Image Segmentation

Tian, Guiyu; Wang, Shuai; Feng, Jie; Zhou, Li; Mu, Yadong

doi:10.1145/3394171.3413990

Cited by 13 publications

(6 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zero-shot image segmentation [2,11,13,17,28,33,35,41] aims at segmenting an image containing classes that are not seen during the training phase. The alignment between visual embeddings and text embeddings of categories is of great importance in this task.…”

Section: Related Work 21 Zero-shot Image Segmentationmentioning

confidence: 99%

Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation

Wang,

Huang,

Gao

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Traditional 3D segmentation methods can only recognize a fixed range of classes that appear in the training set, which limits their application in real-world scenarios due to the lack of generalization ability. Large-scale visual-language pre-trained models, such as CLIP, have shown their generalization ability in the zero-shot 2D vision tasks, but are still unable to be applied to 3D semantic segmentation directly. In this work, we focus on zero-shot point cloud semantic segmentation and propose a simple yet effective baseline to transfer the visual-linguistic knowledge implied in CLIP to point cloud encoder at both feature and output levels. Both feature-level and output-level alignments are conducted between 2D and 3D encoders for effective knowledge transfer. Concretely, a Multi-granularity Cross-modal Feature Alignment (MCFA) module is proposed to align 2D and 3D features from global semantic and local position perspectives for feature-level alignment. For the output level, per-pixel pseudo labels of unseen classes are extracted using the pre-trained CLIP model as supervision for the 3D segmentation model to mimic the behavior of the CLIP image encoder. Extensive experiments are conducted on two popular benchmarks of point cloud segmentation. Our method outperforms significantly previous state-of-the-art methods under zero-shot setting (+29.2% mIoU on SemanticKITTI and 31.8% mIoU on nuScenes), and further achieves promising results in the annotation-free point cloud semantic segmentation setting, showing its great potential for labelefficient learning.

show abstract

Section: Related Work 21 Zero-shot Image Segmentationmentioning

confidence: 99%

Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation

Wang,

Huang,

Gao

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…[5] used a state of the art segmentation network (DeepLab), and propagate information of unseen classes to pixel embeddings using Word2Vec, together with self-training. [41] leveraged image captions in order to segment unknown classes. [26] learns a generator to produce visual features from semantic word embeddings, similar to [5], but it alternated between generating "good features", while maintaining the structural-relations between categories in the text latent space, as before.…”

Section: Related Workmentioning

confidence: 99%

Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples

Zabari¹,

Hoshen²

2021

Preprint

View full text Add to dashboard Cite

Semantic segmentation is a key computer vision task that has been actively researched for decades. In recent years, supervised methods have reached unprecedented accuracy, however they require many pixel-level annotations for every new class category which is very time-consuming and expensive. Additionally, the ability of current semantic segmentation networks to handle a large number of categories is limited. That means that images containing rare class categories are unlikely to be well segmented by current methods. In this paper we propose a novel approach for creating semantic segmentation masks for every object, without the need for training segmentation networks or seeing any segmentation masks. Our method takes as input the image-level labels of the class categories present in the image; they can be obtained automatically or manually. We utilize a vision-language embedding model (specifically CLIP) to create a rough segmentation map for each class, using model interpretability methods. We refine the maps using a test-time augmentation technique. The output of this stage provides pixel-level pseudo-labels, instead of the manual pixel-level labels required by supervised methods. Given the pseudo-labels, we utilize single-image segmentation techniques to obtain high-quality output segmentation masks. Our method is shown quantitatively and qualitatively to outperform methods that use a similar amount of supervision. Our results are particularly remarkable for images containing rare categories.

show abstract

“…Recent works have explored zero-shot object detection by learning to distinguish between background and novel object regions [24,25], synthesizing unseen class features [26] or using richer textual descriptions [56]. For pixel-level mask prediction, [57][58][59][60][61][62][63][64] perform zero-shot semantic segmentation while [27] tackles the challenging zero-shot instance segmentation task. Since these zero-shot methods only have access to base class annotations, they perform poorly on novel classes.…”

Section: Related Workmentioning

confidence: 99%

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Huynh¹,

Kuen²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.

show abstract

Cap2Seg: Inferring Semantic and Spatial Context from Captions for Zero-Shot Image Segmentation

Cited by 13 publications

References 38 publications

Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation

Transferring CLIP's Knowledge into Zero-Shot Point Cloud Semantic Segmentation

Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Contact Info

Product

Resources

About