AMS-SFE: Towards an Alignment of Manifold Structures via Semantic Feature Expansion for Zero-shot Learning

Guo, Jingcai; Guo, Song

doi:10.1109/icme.2019.00021

Cited by 5 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In machine learning, this process is considered as the problem of Zero-shot Learning [27]. The settings of zero-shot learning can be regarded as an extreme case of transfer learning: the model is trained to imitate human ability in recognizing examples of unseen classes that are not shown during training stage [28][29][30][31][32][33][34][35][36][37]. In conventional supervised learning, the training and testing examples belong to the same class-set, which means that the learned model has already seen some examples of all the classes it encoun- all classes is impossible in real-world applications.…”

Section: Overviewmentioning

confidence: 99%

“…In conventional supervised learning, the training and testing examples belong to the same class-set, which means that the learned model has already seen some examples of all the classes it encoun- all classes is impossible in real-world applications. As such, the zero-shot learning has received increasing attention in recent years [28][29][30][31][32][33][34][35][36][37].…”

Section: Overviewmentioning

confidence: 99%

“…ZSL can be roughly divided into three categories according to the mapping direction that adopted as visual to semantic mapping, semantic to visual mapping, and intermedia mapping (metric learning). Most existing methods map the visual features to semantic feature space spanned by class descriptions and then perform nearest neighbor search [27,28,35,37,49,54,55,69,72,82,122,124]. For example, DeViSE [28] trains a linear mapping function between visual and semantic feature spaces by an effective ranking loss formulation.…”

Section: Existing Workmentioning

confidence: 99%

“…SGAL [92] uses the variational autoencoder with class-specific multi-modal prior to learn the conditional distribution of seen and unseen classes. AMS-SFE [37,49] proposes to expand the semantic feature space and implicitly to align the visual and semantic feature spaces.…”

Section: Existing Workmentioning

confidence: 99%

See 3 more Smart Citations

Learning Robust Visual-semantic Mapping for Zero-shot Learning

Guo

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Zero-shot learning (ZSL) aims at recognizing unseen class examples (e.g., images) with knowledge transferred from seen classes. This is typically achieved by exploiting a semantic feature space shared by both seen and unseen classes, e.g., attributes or word vectors, as the bridge. In ZSL, the common practice is to train a mapping function between the visual and semantic feature spaces with labeled seen class examples. When inferring, given unseen class examples, the learned mapping function is reused to them and recognizes the class labels on some metrics among their semantic relations. However, the visual and semantic feature spaces are generally independent and exist in entirely different manifolds. Under such a paradigm, the ZSL models may easily suffer from the domain shift problem when constructing and reusing the mapping function, which becomes the major challenge in ZSL. In this thesis, we explore effective ways to mitigate the domain shift problem and learn a robust mapping function between the visual and semantic feature spaces. We focus on fully empowering the semantic feature space, which is one of the key building blocks of ZSL.First, we consider to adaptively adjust to rectify the semantic feature space for ZSL. Conventional ZSL models generally regard the semantic feature space unchangeable during training. However, it can be observed that when mapping visual features to semantic feature space, the obtained semantic features are usually overly concentrated. This deficiency affects models' ability to adapt and generalize to more unseen classes. As we know, the process of human beings understanding things is constantly improving. Similarly, we argue the semantic feature space also needs to be dynamically adjusted to accommodate more robust learning of mapping Abstract iii function. Specifically, the adjustment is conducted on both the class prototypes and global distribution during training. Moreover, we also propose to combine the adjustment with a cycle mapping to formulate the training to a more efficient framework that can not only rectify the semantic feature space but also speed up the training process.Second, we consider to align the manifold structures between the visual and semantic feature spaces by expanding the semantic features. Compared to our first method which directly adjusts to rectify the semantic feature space, the expansion process is more conservative and soft. Specifically, we build upon an autoencoderbased model to generate several auxiliary semantic features combining with the previous ones, to expand the space. Additionally, the expansion is jointly guided by an embedded manifold extracted from the visual feature space, which retains its geometrical and structural information. By aligning the two feature spaces, the trained mapping function is more robust and well-matched that significantly mitigates the domain shift problem.Last, we consider to take a further step to explore and empower the correlation between the visual and semantic feature spaces in a more fine-grained pe...

show abstract