Zero-Shot Semantic Segmentation via Variational Mapping

Kato, Naoki; Yamasaki, Toshihiko; Aizawa, Kiyoharu

doi:10.1109/iccvw.2019.00172

Cited by 54 publications

(26 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, it can be trivially transferred between different datasets. Table 6 shows the results on the class split proposed by [49] and [21]. Our method surpass other baselines in both ZSL and GZSL.…”

Section: Experimental Results On Pascal Vocmentioning

confidence: 97%

“…Following previous practice in [6,21,49], let us use the notations S, U for the collections of seen / unseen class names, respectively. S ∩ U = ∅.…”

Section: Our Proposed Model 31 Task Formulationmentioning

confidence: 99%

“…This task is derived from zero shot learning task, yet with a special emphasis of spatial correlation of different image pixels. Only a very limited number of related works have been devoted to this novel task, as can be found in [6,21,49]. Among them, the mainstream approach employs similar idea in general zero-shot learning.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Cap2Seg: Inferring Semantic and Spatial Context from Captions for Zero-Shot Image Segmentation

Tian

Wang

Feng

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Zero-shot image segmentation refers to the task of segmenting pixels from specific unseen semantic class. Previous methods mainly rely on historic segmentation tasks, such as using semantic embedding or word embedding of class names to infer a new segmentation model. In this work we describe Cap2Seg, a novel solution of zero-shot image segmentation that harnesses accompanying image captions for intelligently inferring spatial and semantic context for the zero-shot image segmentation task. As our main insight, image captions often implicitly entail the occurrence of a new class in an image and its most-confident spatial distribution. We define a contextual entailment question (CEQ) that tailors BERT-like text models. In specific, the proposed networks for inferring unseen classes consists of three branches (global / local / semi-global), which infer labels of unseen class from image level, image-stripe level or pixel level respectively. Comprehensive experiments and ablation studies are conducted on two image benchmarks, COCO-stuff and Pascal VOC. All clearly demonstrate the effectiveness of the proposed Cap2Seg, including a set of hardest unseen classes (i.e., image captions do not literally contain the class names and direct matching for inference fails). CCS CONCEPTS • Computing methodologies → Image segmentation.

show abstract

Section: Experimental Results On Pascal Vocmentioning

confidence: 97%

“…Following previous practice in [6,21,49], let us use the notations S, U for the collections of seen / unseen class names, respectively. S ∩ U = ∅.…”

Section: Our Proposed Model 31 Task Formulationmentioning

confidence: 99%

See 1 more Smart Citation

Cap2Seg: Inferring Semantic and Spatial Context from Captions for Zero-Shot Image Segmentation

Tian

Wang

Feng

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…In contrast, due to the uniqueness of semantic segmentation, we utilize pixel-wise contextual information to generate pixel-wise features. Zero-shot Semantic Segmentation: The term zero-shot semantic segmentation appeared in prior works [3,17,43,49], in which only SPNet [43] and ZS3Net [3] focused on multi-category semantic segmentation. SPNet achieves knowledge transfer between seen and unseen categories via semantic projection layer and calibration method, while ZS3Net aims to generate pixel-wise features to finetune the classifier, which is biased to the seen categories.…”

Section: Related Workmentioning

confidence: 99%

“…To the best of our knowledge, there are quite few works on zeroshot semantic segmentation [3,17,43,49], in which only SPNet [43] and ZS3Net [3] can segment an image with multiple categories. SP-Net extends a segmentation network by projecting visual features to semantic word embeddings.…”

Section: Introductionmentioning

confidence: 99%

Context-aware Feature Generation For Zero-shot Semantic Segmentation

Zhou

Niu

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

103

View full text Add to dashboard Cite

Existing semantic segmentation models heavily rely on dense pixelwise annotations. To reduce the annotation pressure, we focus on a challenging task named zero-shot semantic segmentation, which aims to segment unseen objects with zero annotations. This task can be accomplished by transferring knowledge across categories via semantic word embeddings. In this paper, we propose a novel context-aware feature generation method for zero-shot segmentation named CaGNet. In particular, with the observation that a pixel-wise feature highly depends on its contextual information, we insert a contextual module in a segmentation network to capture the pixel-wise contextual information, which guides the process of generating more diverse and context-aware features from semantic word embeddings. Our method achieves state-of-the-art results on three benchmark datasets for zero-shot segmentation. CCS CONCEPTS • Computing methodologies → Image segmentation.

show abstract