VisualSem: A High-quality Knowledge Graph for Vision and Language

Alberts, Houda; Huang, Teresa; Deshpande, Yash; Liu, Yibo; Cho, Kyunghyun; Vania, Clara; Calixto, Iacer

doi:10.48550/arxiv.2008.09150

Cited by 4 publications

(8 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Three knowledge graph datasets are adopted in the pre-training process. Visu-alSem [2] is a high-quality multi-modal knowledge graph dataset for vision and language concepts, including entities with multilingual glosses, multiple illustrative images, and visually relevant relations, covering a total number of 90k nodes, 1.3M glosses and 938k images. 13 semantic relations are used to connect different entities in the graph, while the entities in VisualSem are linked to Wikipedia articles, WordNet [34], and high-quality images from ImageNet [10].…”

Section: Methods Flickr30k (1k Test Set) Mscoco(5k Test Set) Text Ret...mentioning

confidence: 99%

“…To better fit our pre-training framework, we convert the original image-text pair into the form of triplets, with specifically designed relations 'image of' and 'caption of'. (2) We also use the original CLIP model as the teacher, and use an auxiliary loss L KD to measure the KL distance between the output of CLIP and our model.…”

Section: Training Targetsmentioning

confidence: 99%

“…These advantages over regular-structured data are favorable on various tasks including question-answering [18,70], relation prediction [29,43] and knowledge reasoning [6,59]. In recent years, knowledge graph has also been investigated in the field of computer vision, e.g., scene graph [65], and the integration of both language and image [2]. This bridges the gap between different modalities in the knowledge graph, which inspires us to explore a new knowledge-based pre-training framework, and inject semantic information into simple image-text pairs.…”

Section: Introductionmentioning

confidence: 99%

“…We train our model on three knowledge graph datasets, namely Visual-Genome [24] (scene graph), ConceptNet [46] (language-based graph), and VisualSem [2] (multi-modal graph), and also adopt part of datasets from CLIP to avoid the model forgetting problem. With the knowledge-enhanced pre-training, Knowledge-CLIP achieves consistent improvements over the original CLIP models on various vision and language downstream tasks.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Contrastive Language-Image Pre-Training with Knowledge Graphs

Pan¹,

Tian-zhu²,

Han³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model [38]. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.

show abstract

Section: Methods Flickr30k (1k Test Set) Mscoco(5k Test Set) Text Ret...mentioning

confidence: 99%

Section: Training Targetsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Contrastive Language-Image Pre-Training with Knowledge Graphs

Pan¹,

Tian-zhu²,

Han³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In Tab. 1, we list mainstream multimodal knowledge graph datasets [1,33,43,54,79,85,87,97], constructed by texts and images with detailed information. For instance, VisualGenome [33] is a multimodal knowledge graph which contains 40,480 relations, 108,077 image nodes with objects.…”

Section: Multimodel Knowledge Graphmentioning

confidence: 99%

UKnow: A Unified Knowledge Protocol for Common-Sense Reasoning and Vision-Language Pre-training

Gong¹,

Xie²,

Feng³

et al. 2023

Preprint

View full text Add to dashboard Cite

This work presents a unified knowledge protocol, called UKnow, which facilitates knowledge-based studies from the perspective of data. Particularly focusing on visual and linguistic modalities, we categorize data knowledge into five unit types, namely, in-image, intext, cross-image, cross-text, and image-text. Following this protocol, we collect, from public international news, a large-scale multimodal knowledge graph dataset that consists of 1,388,568 nodes (with 571,791 visionrelated ones) and 3,673,817 triplets. The dataset is also annotated with rich event tags, including 96 coarse labels and 9,185 fine labels, expanding its potential usage. To further verify that UKnow can serve as a standard protocol, we set up an efficient pipeline to help reorganize existing datasets under UKnow format. Finally, we benchmark the performance of some widely-used baselines on the tasks of common-sense reasoning and visionlanguage pre-training. Results on both our new dataset and the reformatted public datasets demonstrate the effectiveness of UKnow in knowledge organization and method evaluation. Code, dataset, conversion tool, and baseline models will be made public. (https://github. com/Gongggg/UKnow)

show abstract

Beyond Entities: A Large-Scale Multi-Modal Knowledge Graph with Triplet Fact Grounding

Liu,

Zhang,

et al. 2024

AAAI

View full text Add to dashboard Cite

Much effort has been devoted to building multi-modal knowledge graphs by visualizing entities on images, but ignoring the multi-modal information of the relation between entities. Hence, in this paper, we aim to construct a new large-scale multi-modal knowledge graph with triplet facts grounded on images that reflect not only entities but also their relations. To achieve this purpose, we propose a novel pipeline method, including triplet fact filtering, image retrieving, entity-based image filtering, relation-based image filtering, and image clustering. In this way, a multi-modal knowledge graph named ImgFact is constructed, which contains 247,732 triplet facts and 3,730,805 images. In experiments, the manual and automatic evaluations prove the reliable quality of our ImgFact. We further use the obtained images to enhance model performance on two tasks. In particular, the model optimized by our ImgFact achieves an impressive 8.38% and 9.87% improvement over the solutions enhanced by an existing multi-modal knowledge graph and VisualChatGPT on F1 of relation classification. We release ImgFact and its instructions at https://github.com/kleinercubs/ImgFact.

show abstract

VisualSem: A High-quality Knowledge Graph for Vision and Language

Cited by 4 publications

References 16 publications

Contrastive Language-Image Pre-Training with Knowledge Graphs

Contrastive Language-Image Pre-Training with Knowledge Graphs

UKnow: A Unified Knowledge Protocol for Common-Sense Reasoning and Vision-Language Pre-training

Beyond Entities: A Large-Scale Multi-Modal Knowledge Graph with Triplet Fact Grounding

Contact Info

Product

Resources

About