Learning Graph Embeddings for Compositional Zero-shot Learning

Abstract

In compositional zero-shot learning, the goal is to recognize unseen compositions (e.g. old dog) of observed visual primitives states (e.g. old, cute) and objects (e.g. car, dog) in the training set. This is challenging because the same state can for example alter the visual appearance of a dog drastically differently from a car. As a solution, we propose a novel graph formulation called Compositional Graph Embedding (CGE) that learns image features, compositional classifiers and latent representations of visual primitives in an end-to-end manner. The key to our approach is exploiting the dependency between states, objects and their compositions within a graph structure to enforce the relevant knowledge transfer from seen to unseen compositions. By learning a joint compatibility that encodes semantics between concepts, our model allows for generalization to unseen compositions without relying on an external knowledge base like WordNet. We show that in the challenging generalized compositional zero-shot setting our CGE significantly outperforms the state of the art on MIT-States and UT-Zappos. We also propose a new benchmark for this task based on the recent GQA dataset.

Learning Graph Embeddings for Open World Compositional Zero-Shot Learning

“…MIT-States instead contains images collected through older search engine with limited human annotation leading to significant label noise [35]. To address the limitations of these two datasets, in our previous work [5] we introduced a split built on top of Stanford GQA dataset [59], i.e. the Compositional GQA (C-GQA) dataset.…”

“…We compare with four state-of-the-art methods, Attribute as Operators (AOP) [2], considering objects as vectors and states as matrices modifying them [2]; LabelEm-bed+ (LE+) [1], [2] training a classifier merging state and object embeddings with an MLP; Task-Modular Neural Networks (TMN) [3], modifying the classifier through a gating function receiving as input the queried state-object composition; and SymNet [4], learning object embeddings showing symmetry under different state-based transformations. We also compare Co-CGE with our previous works, CGE [5] and CompCos [10]. We train each model with their default hyperparameters, reporting the closed and open world results of the models with the best AUC on the validation set.…”

“…to produce bounded scores and it is beneficial to avoid prediction to be influenced by the higher magnitude of scores for seen training classes [54] while generalizing better to new ones [55]. This is a greater challenge for our model compared to CGE [5] since we tailor it for the open world. Finally, we learn the mappings φ and ω by minimizing the cross-entropy loss over the cosine logits.…”

“…Our contributions are as follows: (1) We introduce Co-CGE, a graph formulation for the new OW-CZSL problem with an integrated feasibility estimation mechanism used to weight the graph connections; (2) We exploit the dependency between visual primitives and their compositional classes and propose a multimodal compatibility learning framework that embeds related states, objects and their compositions into a shared embedding space learned through cosine logits and feasibility-based margins; (3) We improve the state-of-the-art on MIT states [6], UT Zappos [9] and the recently proposed C-GQA [5] benchmarks on both CZSL and OW-CZSL.…”

“…This means that the embeddings as well as the feasibility scores are influenced by all other composition in the search space, rather than considered in isolation as it was the case in [10]. We extend our OW-CZSl benchmark proposed in [10] to the new C-GQA dataset proposed in [5] with 453 states, 870 objects thus a total OW-CZSL search space of almost 400k compositions. Finally, we significantly improve the state of the art on the challenging OW-CZSL setting.…”