Zero-Shot Visual Question Answering Using Knowledge Graph

Chen, Zhuo; Chen, Jiaoyan; Geng, Yuxia; Pan, Jeff Z.; Yuan, Zonggang; Chen, Huajun

doi:10.1007/978-3-030-88361-4_9

Cited by 44 publications

(39 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The IsA relation is used to represent hyponyms and hypernyms. ConceptNet is often used in a similar way as WordNet: a sub-graph which acts as class hierarchies for augmenting ZSL [38,41,127,128,152,214,217] and FSL [207,214,217] is extracted via matching classes to entities and selecting neighbouring entities. It is also mostly applied in CV tasks but has also been explored in open information extraction.…”

Section: Sub-kg Extractionmentioning

confidence: 99%

“…• DBpedia is also a large-scale general purpose KG whose knowledge are mainly from Wikipedia [6]. It has also been used to augment ZSL, often acting as a complement of relational facts and literals such as entity descriptions [41,62,63]. DBpedia entities are often retrieved via its lookup service which is based on a lexical index built on entity labels and descriptions 5 , while the entity associated facts and literals can be accessed from a DBpedia dump or its online SPARQL endpoint 6 .…”

Section: Sub-kg Extractionmentioning

confidence: 99%

“…Luo et al [111] extracted a sub-KG for object relationships from Visual Genome -a knowledge base that stores connections between image visual concepts and language concepts [93], for augmenting zero-shot object recognition. The KG for augmenting zero-shot VQA in [41] also includes facts from WebChild which is a large collection of commonsense knowledge from the Web [169], besides knowledge from ConceptNet and DBpedia. Zhou et al [230] trained their zero-shot question answering model with facts extracted from WorldTree (V2.0) [202] -a knowledge base that contains explanations for multiple-choice science questions in the form of graph, covering both commonsense and scientific knowledge.…”

Section: Sub-kg Extractionmentioning

confidence: 99%

“…Therefore, some studies proposed to integrate knowledge extracted from different KGs or/and other resources for building a high quality task-specific KG. For example, Chen et al [41] extracted RDF facts from three KGs -ConceptNet, WebChild and DBpedia to generate a unified commonsense KG for augmenting zero-shot VQA; Geng et al [62,63] integrated class hierarchies from WordNet, relational facts and literals from DBpedia, and knowledge transformed from class attributes for constructing KGs for zero-shot image classification; Chen et al [36] considered and integrated class hierarchies from WordNet, and class co-occurrence relations extracted from class attributes and samples for a KG for zero-shot ingredient recognition from food images. Very recently, Geng et al [64] proposed a benchmarking study, where six KG equipped ZSL benchmarks were created for three different tasks and used for evaluating different methods under different auxiliary information settings.…”

Section: Knowledge Integrationmentioning

confidence: 99%

“…Input Mapping [34,84,100,102,103,105,114,132] Class Mapping [1,2,27,127,158] Joint Mapping [38,41,70,83,114,149,152] Data Augmentation…”

Section: Mapping-basedmentioning

confidence: 99%

See 4 more Smart Citations

Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Survey

Chen¹,

Geng²,

Chen³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning methods especially deep neural networks have achieved great success but many of them often rely on a number of labeled samples for training. In real-world applications, we often need to address sample shortage due to e.g., dynamic contexts with emerging prediction targets and costly sample annotation. Therefore, low-resource learning, which aims to learn robust prediction models with no enough resources (especially training samples), is now being widely investigated. Among all the low-resource learning studies, many prefer to utilize some auxiliary information in the form of Knowledge Graph (KG), which is becoming more and more popular for knowledge representation, to reduce the reliance on labeled samples. In this survey, we very comprehensively reviewed over 90 papers about KG-aware research for two major low-resource learning settings -zero-shot learning (ZSL) where new classes for prediction have never appeared in training, and few-shot learning (FSL) where new classes for prediction have only a small number of labeled samples that are available. We first introduced the KGs used in ZSL and FSL studies as well as the existing and potential KG construction solutions, and then systematically categorized and summarized KG-aware ZSL and FSL methods, dividing them into different paradigms such as the mapping-based, the data augmentation, the propagation-based and the optimization-based. We next presented different applications, including not only KG augmented prediction tasks in Computer Vision and Natural Language Processing (e.g., image classification, visual question answering, text classification and knowledge extraction), but also tasks for KG curation (e.g., inductive KG completion), and some typical evaluation resources for each task. We eventually discussed some challenges and future directions on aspects such as new learning and reasoning paradigms, and the construction of high quality KGs.

show abstract

Section: Sub-kg Extractionmentioning

confidence: 99%

Section: Sub-kg Extractionmentioning

confidence: 99%

Section: Sub-kg Extractionmentioning

confidence: 99%

Section: Knowledge Integrationmentioning

confidence: 99%

“…Input Mapping [34,84,100,102,103,105,114,132] Class Mapping [1,2,27,127,158] Joint Mapping [38,41,70,83,114,149,152] Data Augmentation…”

Section: Mapping-basedmentioning

confidence: 99%

See 3 more Smart Citations

Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Survey

Chen¹,

Geng²,

Chen³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

A review on multimodal zero‐shot learning

Cao

Sun

et al. 2023

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Multimodal learning provides a path to fully utilize all types of information related to the modeling target to provide the model with a global vision. Zero-shot learning (ZSL) is a general solution for incorporating prior knowledge into data-driven models and achieving accurate class identification. The combination of the two, known as multimodal ZSL (MZSL), can fully exploit the advantages of both technologies and is expected to produce models with greater generalization ability. However, the MZSL algorithms and applicationshave not yet been thoroughly investigated and summarized. This study fills this gap by providing an objective overview of MZSL's definition, typical algorithms, representative applications, and critical issues. This article will not only provide researchers in this field with a comprehensive perspective, but it will also highlight several promising research directions.

show abstract