Gaussian Visual-Linguistic Embedding for Zero-Shot Recognition

Mukherjee, Tanmoy; Hospedales, Timothy M.

doi:10.18653/v1/d16-1089

Cited by 43 publications

(30 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that existing works follow two settings. Some of them [30,18] Comparative results on ILSVRC 2012/2010 Even fewer published results on this dataset are available. Table 3 shows that our model clearly outperform the state-of-the-art alternatives by a large margin.…”

Section: Experiments On Imagenetmentioning

confidence: 99%

Learning a Deep Embedding Model for Zero-Shot Learning

Zhang

Xiang

Gong

2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

671

594

View full text Add to dashboard Cite

Zero-shot learning (ZSL) models rely on learning a joint embedding space where both textual/semantic description of object classes and visual representation of object images can be projected to for nearest neighbour search. Despite the success of deep neural networks that learn an end-toend model between text and images in other vision problems such as image captioning, very few deep ZSL model exists and they show little advantage over ZSL models that utilise deep feature representations but do not learn an end-to-end embedding. In this paper we argue that the key to make deep ZSL models succeed is to choose the right embedding space. Instead of embedding into a semantic space or an intermediate space, we propose to use the visual space as the embedding space. This is because that in this space, the subsequent nearest neighbour search would suffer much less from the hubness problem and thus become more effective. This model design also provides a natural mechanism for multiple semantic modalities (e.g., attributes and sentence descriptions) to be fused and optimised jointly in an end-to-end manner. Extensive experiments on four benchmarks show that our model significantly outperforms the existing models.

show abstract

Section: Experiments On Imagenetmentioning

confidence: 99%

Learning a Deep Embedding Model for Zero-Shot Learning

Zhang

Xiang

Gong

2017

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

671

594

View full text Add to dashboard Cite

show abstract

“…Beyond injective embedding: Similar to our motivation, some attempts have been made to go beyond the injective mapping. One approach is to design the embedding function to be stochastic and map an instance to a certain probability distribution (e.g., Gaussian) instead of a single point [43,38,39]. However, learning distributions is typically difficult/expensive and often lead to approximate solutions such as Monte Carlo sampling.…”

Section: Related Workmentioning

confidence: 99%

Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval

Song

Soleymani

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

238

129

View full text Add to dashboard Cite

Visual-semantic embedding aims to find a shared latent space where related visual and textual instances are close to each other. Most current methods learn injective embedding functions that map an instance to a single point in the shared space. Unfortunately, injective embedding cannot effectively handle polysemous instances with multiple possible meanings; at best, it would find an average representation of different meanings. This hinders its use in real-world scenarios where individual instances and their cross-modal associations are often ambiguous. In this work, we introduce Polysemous Instance Embedding Networks (PIE-Nets) that compute multiple and diverse representations of an instance by combining global context with locally-guided features via multi-head self-attention and residual learning. To learn visual-semantic embedding, we tie-up two PIE-Nets and optimize them jointly in the multiple instance learning framework. Most existing work on cross-modal retrieval focuses on image-text data. Here, we also tackle a more challenging case of video-text retrieval. To facilitate further research in video-text retrieval, we release a new dataset of 50K video-sentence pairs collected from social media, dubbed MRW (my reaction when). We demonstrate our approach on both image-text and video-text retrieval scenarios using MS-COCO, TGIF, and our new MRW dataset.

show abstract

“…[54] proposes to combine the semantic embedding and knowledge graph with graph convolutional network [24]. An orthogonal direction is generative model [52,38], where class-conditional distribution is learned based on the Gaussian assumption.…”

Section: Related Workmentioning

confidence: 99%

F-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning

Xian

Sharma

Schiele³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

487

358

View full text Add to dashboard Cite

When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zeroand few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.

show abstract

Gaussian Visual-Linguistic Embedding for Zero-Shot Recognition

Cited by 43 publications

References 19 publications

Learning a Deep Embedding Model for Zero-Shot Learning

Learning a Deep Embedding Model for Zero-Shot Learning

Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval

F-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning

Contact Info

Product

Resources

About