Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Abstract

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.

Localized Vision-Language Matching for Open-vocabulary Object Detection

“…To learn the semantics of novel classes, recent methods [3,13,16,41,44] have simplified the problem by providing image-caption pairs as a weak supervision signal. Such pairs are cheap to acquire and make the problem tractable.…”

“…Most of these methods require big dataset with millions of image-caption pairs to train such a model. They either use this model to align image-regions with captions and generate object-box pseudo labels [16,44] or as region-image feature extractor to classify the regions [13]. Many weakly-supervised [1,3,7,34,43] approaches have been proposed to perform such object grounding.…”

“…CLIP (cropped reg) [13] uses the CLIP pre-trained model on 400M image-caption pairs on object proposals obtained by an object detector trained on known classes. XP-Mask [16] first learns a class-agnostic region proposal and segmentation model from the known classes and then uses this model as a teacher to generate pseudo masks for self-training a student model. Finally, we also compare with VILD [13] which uses CLIP soft predictions to distil semantic information and train an object detector.…”