A Novel Perspective to Zero-Shot Learning: Towards an Alignment of Manifold Structures via Semantic Feature Expansion

Guo, Jingcai; Guo, Song

doi:10.1109/tmm.2020.2984091

Cited by 32 publications

(14 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This task aims at recognizing unseen categories with the help of seen categories and their semantic description [33], [34], [35], [36], [37], [38], [39], [40], [41], [42]. In the traditional setting, the model is trained on the seen categories' images and evaluated on the images of unseen ones.…”

Section: B Zero-shot Learning (Zsl)mentioning

confidence: 99%

Relation-aware Compositional Zero-shot Learning for Attribute-Object Pair Recognition

Xu,

Wang,

Wong

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper proposes a novel model for recognizing images with composite attribute-object concepts, notably for composite concepts that are unseen during model training. We aim to explore the three key properties required by the taskrelation-aware, consistent, and decoupled -to learn rich and robust features for primitive concepts that compose attributeobject pairs. To this end, we propose the Blocked Message Passing Network (BMP-Net). The model consists of two modules. The concept module generates semantically meaningful features for primitive concepts, whereas the visual module extracts visual features for attributes and objects from input images. A message passing mechanism is used in the concept module to capture the relations between primitive concepts. Furthermore, to prevent the model from being biased towards seen composite concepts and reduce the entanglement between attributes and objects, we propose a blocking mechanism that equalizes the information available to the model for both seen and unseen concepts. Extensive experiments and ablation studies on two benchmarks show the efficacy of the proposed model.

show abstract

Section: B Zero-shot Learning (Zsl)mentioning

confidence: 99%

Relation-aware Compositional Zero-shot Learning for Attribute-Object Pair Recognition

Xu,

Wang,

Wong

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Such kind of word embeddings are widely used in natural language processing problems and can be ef-ficiently extended to zero-shot learning. Among them, word2vec [42,43], FastText [44] and GloVe [45] vectors are most frequently used [46][47][48][49]. Now, we elaborate on different categories of semantic features as follows.…”

Section: Categories Of Semanticsmentioning

confidence: 99%

“…ZSL can be roughly divided into three categories according to the mapping direction that adopted as visual to semantic mapping, semantic to visual mapping, and intermedia mapping (metric learning). Most existing methods map the visual features to semantic feature space spanned by class descriptions and then perform nearest neighbor search [27,28,35,37,49,54,55,69,72,82,122,124]. For example, DeViSE [28] trains a linear mapping function between visual and semantic feature spaces by an effective ranking loss formulation.…”

Section: Existing Workmentioning

confidence: 99%

“…frequently, yet to obtain their annotations is expensive and time-consuming. Zeroshot learning (ZSL), which aims to imitate human ability in recognizing unseen classes under such restrictions, has received increasing attention in the most recent years [27,28,30,35,49]. ZSL takes utilization of seen classes with labeled examples and auxiliary knowledge that can be shared and transferred between seen and unseen classes to solve the problem.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, several methods have been proposed to mitigate the domain shift problem. For example, some methods consider directly doing the alignment between these two feature spaces when constructing the mapping function [49,69,124]. Differently, some methods try to synthesize unseen class examples/features based on generative adversarial networks (GANs) or variational autoencoders (VAEs) and involve them in the training process [66,67,125].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning Robust Visual-semantic Mapping for Zero-shot Learning

Guo

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Zero-shot learning (ZSL) aims at recognizing unseen class examples (e.g., images) with knowledge transferred from seen classes. This is typically achieved by exploiting a semantic feature space shared by both seen and unseen classes, e.g., attributes or word vectors, as the bridge. In ZSL, the common practice is to train a mapping function between the visual and semantic feature spaces with labeled seen class examples. When inferring, given unseen class examples, the learned mapping function is reused to them and recognizes the class labels on some metrics among their semantic relations. However, the visual and semantic feature spaces are generally independent and exist in entirely different manifolds. Under such a paradigm, the ZSL models may easily suffer from the domain shift problem when constructing and reusing the mapping function, which becomes the major challenge in ZSL. In this thesis, we explore effective ways to mitigate the domain shift problem and learn a robust mapping function between the visual and semantic feature spaces. We focus on fully empowering the semantic feature space, which is one of the key building blocks of ZSL.First, we consider to adaptively adjust to rectify the semantic feature space for ZSL. Conventional ZSL models generally regard the semantic feature space unchangeable during training. However, it can be observed that when mapping visual features to semantic feature space, the obtained semantic features are usually overly concentrated. This deficiency affects models' ability to adapt and generalize to more unseen classes. As we know, the process of human beings understanding things is constantly improving. Similarly, we argue the semantic feature space also needs to be dynamically adjusted to accommodate more robust learning of mapping Abstract iii function. Specifically, the adjustment is conducted on both the class prototypes and global distribution during training. Moreover, we also propose to combine the adjustment with a cycle mapping to formulate the training to a more efficient framework that can not only rectify the semantic feature space but also speed up the training process.Second, we consider to align the manifold structures between the visual and semantic feature spaces by expanding the semantic features. Compared to our first method which directly adjusts to rectify the semantic feature space, the expansion process is more conservative and soft. Specifically, we build upon an autoencoderbased model to generate several auxiliary semantic features combining with the previous ones, to expand the space. Additionally, the expansion is jointly guided by an embedded manifold extracted from the visual feature space, which retains its geometrical and structural information. By aligning the two feature spaces, the trained mapping function is more robust and well-matched that significantly mitigates the domain shift problem.Last, we consider to take a further step to explore and empower the correlation between the visual and semantic feature spaces in a more fine-grained pe...

show abstract