VisualSem: a high-quality knowledge graph for vision and language

Alberts, Houda; Huang, Teresa; Deshpande, Yash; Liu, Yibo; Cho, Kyunghyun; Vania, Clara; Calixto, Iacer

doi:10.18653/v1/2021.mrl-1.13

Cited by 19 publications

(6 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Besides, by merging the relations with similar semantics into the unified commonsense relation set, these relations have a large semantic distance from each other and are stable for the task of visual explicit relational inference. Besides, commonsense KGs are also widely applied for crossmodal structured knowledge retrieval on multi-modal tasks, for example, SGG [40], Visual Commonsense Reasoning [41], Image Captioning [42], and Multimodal Retrieval [43].…”

Section: Knowledge Graphmentioning

confidence: 99%

DSGEM: Dual scene graph enhancement module‐based visual question answering

Wang

et al. 2023

IET Computer Vision

View full text Add to dashboard Cite

Visual Question Answering (VQA) aims to appropriately answer a text question by understanding the image content. Attention‐based VQA models mine the implicit relationships between objects according to the feature similarity, which neglects the explicit relationships between objects, for example, the relative position. Most Visual Scene Graph‐based VQA models exploit the relative positions or visual relationships between objects to construct the visual scene graph, while they suffer from the semantic insufficiency of visual edge relations. Besides, the scene graph of text modality is often ignored in these works. In this article, a novel Dual Scene Graph Enhancement Module (DSGEM) is proposed that exploits the relevant external knowledge to simultaneously construct two interpretable scene graph structures of image and text modalities, which makes the reasoning process more logical and precise. Specifically, the authors respectively build the visual and textual scene graphs with the help of commonsense knowledge and syntactic structure, which explicitly endows the specific semantics to each edge relation. Then, two scene graph enhancement modules are proposed to propagate the involved external and structural knowledge to explicitly guide the feature interaction between objects (nodes). Finally, the authors embed such two scene graph enhancement modules to existing VQA models to introduce the explicit relation reasoning ability. Experimental results on both VQA V2 and OK‐VQA datasets show that the proposed DSGEM is effective and compatible to various VQA architectures.

show abstract

Section: Knowledge Graphmentioning

confidence: 99%

DSGEM: Dual scene graph enhancement module‐based visual question answering

Wang

et al. 2023

IET Computer Vision

View full text Add to dashboard Cite

show abstract

“…Nous envisageons de modéliser explicitement le lien entre les entités nommées et les images les représentant, en les considérant toutes deux comme des noeuds du graphe de connaissance, en exploitant par exemple les données structurées de Wikidata et Wikimédia Commons. Quelques travaux explorent des approches similaires mais sont limités à de petits graphes de connaissance et une évaluation intrinsèque des représentations en complétant le même graphe (Xie et al, 2017;Pezeshkpour et al, 2018;Wilcke et al, 2020;Alberts et al, 2021).…”

Section: Liens Entre Les Entitésunclassified

Knowledge-based Visual Question Answering about Named Entities

Lerner

2023

SIGIR Forum

View full text Add to dashboard Cite

This thesis is positioned at the intersection of several research fields, Natural Language Processing, Information Retrieval (IR) and Computer Vision, which have unified around representation learning and pre-training methods. We have defined and studied a new multimodal task: Knowledge-based Visual Question Answering about Named Entities (KVQAE). We were particularly interested in cross-modal interactions and different ways of representing named entities. We also focused on data used to train and, more importantly, evaluate Question Answering systems through different metrics. We annotated a dataset for this purpose, the first in KVQAE comprising various types of entities. We also defined an experimental framework for dealing with KVQAE in two stages through an unstructured knowledge base and identified IR as the main bottleneck of KVQAE, especially for questions about non-person entities. To improve the IR stage, we studied different multimodal fusion methods, which are pre-trained through an original task: the Multimodal Inverse Cloze Task. We found that these models leveraged a cross-modal interaction that we had not originally considered, and which may address the heterogeneity of visual representations of named entities. These results were strengthened by a study of the CLIP model, which allows this cross-modal interaction to be modeled directly. Awarded by : Université Paris-Saclay, Orsay, France on 8 November 2023. Supervised by : Olivier Ferret and Camille Guinaudeau. Available at : https://www.theses.fr/s247993.

show abstract

“…We use Multi Modal Knowledge Graph VisualSem (Alberts et al, 2020) to do text entity extraction and grounding to KG for the target complex word. For entity extraction, CLIP textual embedding (Radford et al, 2021) were used as defined in original paper.…”

Section: Knowledge Graphmentioning

confidence: 99%

teamPN at TSAR-2022 Shared Task: Lexical Simplification using Multi-Level and Modular Approach

Nikita,

Rajpoot

2022

Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022)

View full text Add to dashboard Cite

Lexical Simplification is the process of reducing the lexical complexity of a text by replacing difficult words with easier to read (or understand) expressions while preserving the original information and meaning. This paper explains the work done by our team "teamPN" for English track of TSAR 2022 Shared Task of Lexical Simplification. We created a modular pipeline which combines transformers based models with traditional NLP methods like paraphrasing and verb sense disambiguation. We created a multi level and modular pipeline where the target text is treated according to its semantics (Part of Speech Tag). Pipeline is multi level as we utilize multiple source models to find potential candidates for replacement. It is modular as we can switch the source models and their weighting in the final re-ranking.

show abstract

VisualSem: a high-quality knowledge graph for vision and language

Cited by 19 publications

References 31 publications

DSGEM: Dual scene graph enhancement module‐based visual question answering

DSGEM: Dual scene graph enhancement module‐based visual question answering

Knowledge-based Visual Question Answering about Named Entities

teamPN at TSAR-2022 Shared Task: Lexical Simplification using Multi-Level and Modular Approach

Contact Info

Product

Resources

About