Generalized Zero-Shot Recognition Based on Visually Semantic Embedding

Abstract

We propose a novel Generalized Zero-Shot learning (GZSL) method that is agnostic to both unseen images and unseen semantic vectors during training. Prior works in this context propose to map high-dimensional visual features to the semantic domain, which we believe contributes to the semantic gap. To bridge the gap, we propose a novel low-dimensional embedding of visual instances that is "visually semantic." Analogous to semantic data that quantifies the existence of an attribute in the presented instance, components of our visual embedding quantifies existence of a prototypical part-type in the presented instance. In parallel, as a thought experiment, we quantify the impact of noisy semantic data by utilizing a novel visual oracle to visually supervise a learner. These factors, namely semantic noise, visual-semantic gap and label noise lead us to propose a new graphical model for inference with pairwise interactions between label, semantic data, and inputs. We tabulate results on a number of benchmark datasets demonstrating significant improvement in accuracy over state-of-art under both semantic and visual supervision.

Zero-Shot Learning and its Applications from Autonomous Vehicles to COVID-19 Diagnosis: A Review

“…ZSL models can be seen from two points of views in terms of training and test phase: Classic ZSL and Generalised ZSL (GZSL) settings. In the classic ZSL settings, the model only detects the presence of new classes at the test phase, while in GZSL settings, the model predicts both unseen and seen classes at the test time; hence, GZSL is more applicable for realworld scenarios [75,86,94,145,210]. The same idea can be applied to FSL to train in the generalised model, called generalised few-shot learning (GFSL) that detects both known and novel classes at the test time.…”

“…e.g. in [63], [83], [91], [134], [175], [181], [192], [193], [207], [210]. The Euclidean spaces are more conventional and simpler as the data has a flat representation in such spaces.…”

“…DGP [63] proposes dense graph propagation to propagate knowledge directly through dense connections. In [210], a graphical model with a low dimensional visually semantic space is utilised which has a chain-like structure to close the gap between the high-dimensional features and the semantic domain.…”

“…Binary, relative and real-valued [179], Random Forest Model [59], HEX Graph [31] Direct Learning Implicit knowledge representation Linear [6,7,12,17,38,43,114,123,137,171,190,197] or Non-Linear [23,71,84,159,171,185]Compatibility Functions Explicit knowledge representation Graph Conv. Networks (GCN) [181], Knowledge Graphs [63,83,91,134], 3-Node Chains [210], Matrix Tri-Factorisation with Manifold Regularisation [193] Cross-Modal Latent Embedding Fusion-based Models Fusion of seen class data Combination of seen classes properties [22,93,204], Combination of seen class scores [192] Common Representation Space Models…”

Deep Zero-Shot Learning for Scene Sketch

“…To the best of our knowledge, only Ye et al [8] have proposed a deep CNN model "Scene-Net" for scene sketch classification. Existing ZSL methods engineered for scene photo mainly differ in what semantic knowledge are used: typically either word vector [12], attribute [14,15,27] or text description [19]. Sketch is different from photo.…”