Learning Modality-Invariant Latent Representations for Generalized Zero-shot Learning

Abstract

Recently, feature generating methods have been successfully applied to zero-shot learning (ZSL). However, most previous approaches only generate visual representations for zero-shot recognition. In fact, typical ZSL is a classic multi-modal learning protocol which consists of a visual space and a semantic space. In this paper, therefore, we present a new method which can simultaneously generate both visual representations and semantic representations so that the essential multi-modal information associated with unseen classes can be captured. Specifically, we address the most challenging issue in such a paradigm, i.e., how to handle the domain shift and thus guarantee that the learned representations are modalityinvariant. To this end, we propose two strategies: 1) leveraging the mutual information between the latent visual representations and the semantic representations; 2) maximizing the entropy of the joint distribution of the two latent representations. By leveraging the two strategies, we argue that the two modalities can be well aligned. At last, extensive experiments on five widely used datasets verify that the proposed method is able to significantly outperform previous the state-of-the-arts.

Learning Aligned Cross-Modal Representation for Generalized Zero-Shot Classification

“…CADA-VAE (Schönfeld et al 2019) combines cross-modal and cross-reconstruction, and leverages two independent VAEs to align features via minimizing distance between the latent distributions of visual and semantic features. Based on CADA-VAE, Li et al (Li et al 2020) learns modalityinvariant latent representations by maximizing mutual information and entropy on latent space. DE-VAE (Ma and Hu 2020) adopts a deep embedding model to learn the mapping from the semantic space to the visual feature space.…”

Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning

“…Some state-of-the-art GZSL approaches [18], [19], [20], [21], [22], [23] use generative models, such as generative adversarial nets (GANs) [24], variational autoencoders (VAEs) [25], or different variants of hybrid GAN/VAEs, to find an alignment between class-level semantic descriptions and visual representations. As such, the zero-shot learning problem becomes one of a traditional supervised classification task.…”

Attribute-Modulated Generative Meta Learning for Zero-Shot Classification

“…Imrattanatra et al [16] propose a embedding model based on a knowledge graph. Such methods resort to learning a projection from visual space to semantic space [16,17,21,38,53] or the reverse [36,58]. Then, ZSL can be accomplished by ranking similarity or compatibility in the shared space.…”