Zero-Shot Instance Segmentation

Zheng, Ye; Wu, Jiahong; Zhang, Faen; Li, Cui

doi:10.1109/cvpr46437.2021.00262

Cited by 45 publications

(32 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation Metrics. For both detection and segmentation experiments, we report the mean Average Precision (mAP) at intersection-over-union (IoU) of 0.5 following conventional zero-shot settings [24,27,28]. To analyze the performances on base and target classes, we measure the mAP scores in two settings: i) constrained setting where the model is only evaluated on test images belonging to either base classes or target classes; ii) generalized setting in which a model is tested jointly on both base and target class images.…”

Section: Methodsmentioning

confidence: 99%

“…Recent works have explored zero-shot object detection by learning to distinguish between background and novel object regions [24,25], synthesizing unseen class features [26] or using richer textual descriptions [56]. For pixel-level mask prediction, [57][58][59][60][61][62][63][64] perform zero-shot semantic segmentation while [27] tackles the challenging zero-shot instance segmentation task. Since these zero-shot methods only have access to base class annotations, they perform poorly on novel classes.…”

Section: Related Workmentioning

confidence: 99%

“…Different from previous approaches that are limited to classes with mask annotations, zero-shot instance/semantic segmentation aims to segment novel classes without training samples via high-level semantic descriptions such as word embeddings. However, current zero-shot approaches on both object detection [24][25][26] and instance segmentation [27] suffer from low novel-class performances as high-level word embeddings cannot effectively encode fine-grained shape information. To overcome this, the recent OVR [28] work pretrains a visual backbone on captioned images to learn rich visual features.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Huynh¹,

Kuen²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Huynh¹,

Kuen²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…ZSI [98] is a zero-shot model which tackles detection task and segmentation task simultaneous. ZSI consists of three components: zero-shot detector, semantic mask head and synthesized background prediction head.…”

Section: Zs-yolomentioning

confidence: 99%

A Survey of Deep Learning for Low-Shot Object Detection

Huang¹,

Zhang²,

Xue³

et al. 2021

Preprint

View full text Add to dashboard Cite

Object detection is a fundamental task in computer vision and image processing. Current deep learning based object detectors have been highly successful with abundant labeled data. But in real life, it is not guaranteed that each object category has enough labeled samples for training. These large object detectors are easy to overfit when the training data is limited. Therefore, it is necessary to introduce few-shot learning and zeroshot learning into object detection, which can be named low-shot object detection together. Low-Shot Object Detection (LSOD) aims to detect objects from a few or even zero labeled data, which can be categorized into few-shot object detection (FSOD) and zero-shot object detection (ZSD), respectively. This paper conducts a comprehensive survey for deep learning based FSOD and ZSD. First, this survey classifies methods for FSOD and ZSD into different categories and discusses the pros and cons of them. Second, this survey reviews dataset settings and evaluation metrics for FSOD and ZSD, then analyzes the performance of different methods on these benchmarks. Finally, this survey discusses future challenges and promising directions for FSOD and ZSD.

show abstract

“…Zero-shot learning (ZSL) was originally proposed to tackle the specific classification problem, where the model is expected to be capable of classifying the samples belonging to the novel categories which are not seen previously during training. The problem setup has been extended to other applications such as detection [5,20,7] and segmentation [6,31]. Here we provide a brief review of the works of zero-shot classification [14,21,1,2,13,22,29,27,28,22,30,19].…”

Section: Related Workmentioning

confidence: 99%

Make an Omelette with Breaking Eggs: Zero-Shot Learning for Novel Attribute Synthesis

Li¹,

Chao

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

Most of the existing algorithms for zero-shot classification problems typically rely on the attributebased semantic relations among categories to realize the classification of novel categories without observing any of their instances. However, training the zero-shot classification models still requires attribute labeling for each class (or even instance) in the training dataset, which is also expensive. To this end, in this paper, we bring up a new problem scenario: "Are we able to derive zero-shot learning for novel attribute detectors/classifiers and use them to automatically annotate the dataset for labeling efficiency?" Basically, given only a small set of detectors that are learned to recognize some manually annotated attributes (i.e., the seen attributes), we aim to synthesize the detectors of novel attributes in a zero-shot learning manner. Our proposed method, Zero Shot Learning for Attributes (ZSLA), which is the first of its kind to the best of our knowledge, tackles this new research problem by applying the set operations to first decompose the seen attributes into their basic attributes and then recombine these basic attributes into the novel ones. Extensive experiments are conducted to verify the capacity of our synthesized detectors for accurately capturing the semantics of the novel attributes and show their superior performance in terms of detection and localization compared to other baseline approaches. Moreover, with using only 32 seen attributes on the Caltech-UCSD Birds-200-2011 dataset, our proposed method is able to synthesize other 207 novel attributes, while various generalized zero-shot classification algorithms trained upon the dataset re-annotated by our synthesized attribute detectors are able to provide comparable performance with those trained with the manual ground-truth annotations.

show abstract

Zero-Shot Instance Segmentation

Cited by 45 publications

References 23 publications

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

A Survey of Deep Learning for Low-Shot Object Detection

Make an Omelette with Breaking Eggs: Zero-Shot Learning for Novel Attribute Synthesis

Contact Info

Product

Resources

About