TSVFN: Two-Stage Visual Fusion Network for multimodal relation extraction

Zhao, Qihui; Guo, Nan

doi:10.1016/j.ipm.2023.103264

Cited by 25 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Numerous efforts have been devoted to extracting relational facts from multimodal data and developing related datasets. Notably, MNRE [33,34] is the first multimodal relation extraction dataset, with a series of work demonstrating that incorporating multimodal information and efficient alignment strategy for textual and visual representations can effectively boost the performance of relation extraction in social media texts [26,27,29,32]. Wan et al [22] In future work, we will explore ways to enhance the discriminative ability of learned representations [11] and optimally fuse the complementary information of each modal to uncover the intrinsic structure of multimodal relational facts [23,24].…”

Section: Related Workmentioning

confidence: 99%

MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation

He,

Wang,

Cao

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Extracting relational facts from multimodal data is a crucial task in the field of multimedia and knowledge graphs that feeds into widespread real-world applications. The emphasis of recent studies centers on recognizing relational facts in which both entities are present in one modality and supplementary information is used from other modalities. However, such works disregard a substantial amount of multimodal relational facts that arise across different modalities, such as one entity seen in a text and another in an image. In this paper, we propose a new task, namely Multimodal Object-Entity Relation Extraction, which aims to extract "object-entity" relational facts from image and text data. To facilitate research on this task, we introduce MORE, a new dataset comprising 21 relation types and 20,264 multimodal relational facts annotated on 3,559 pairs of textual news titles and corresponding images. To show the challenges of Multimodal Object-Entity Relation Extraction, we evaluated recent state-of-the-art methods for multimodal relation extraction and conducted a comprehensive experimentation analysis on MORE. Our results demonstrate significant challenges for existing methods, underlining the need for further research on this task. Based on our experiments, we identify several promising directions for future research. The MORE dataset and code are available at https://github.com/NJUNLP/MORE.

show abstract

Section: Related Workmentioning

confidence: 99%

MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation

He,

Wang,

Cao

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…Latent Type Classifier 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 2022b; , multimodal relation extraction (Zhao et al, 2023;, and multimodal entity linking (Zhang et al, 2023;Wang et al, 2022c). Information extraction techniques that incorporate multimodality form the foundation for constructing multimodal knowledge bases, providing ample data support for applications such as question-answering systems, information retrieval, and more.…”

Section: Known Type Classifiermentioning

confidence: 99%

Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing

Zhang,

Fan,

Song

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Fine-grained entity typing (FGET) aims to assign appropriate fine-grained types to entity mentions within their context, which is an important foundational task in natural language processing. Previous approaches for FGET only utilized textual context information. However, in the form of short text, the contextual semantic information is often insufficient for FGET. In many real-world scenarios, text is often accompanied by images, and the visual context is valuable for FGET. To this end, we firstly propose a new task called multimodal fine-grained entity typing (MFGET). Then we construct a large-scale dataset for multimodal fine-grained entity typing called MFIGER based on FIGER. To fully leverage both textual and visual information, we propose a novel Multimodal Object-Level Visual Context Network (MOVCNet). MOVCNet can capture fine-grained semantic information by detecting objects in images, and effectively merge both textual and visual context. Experimental results demonstrate that our approach achieves superior classification performance compared to previous approaches.

show abstract

Knowledge Graph for Solubility Big Data: Construction and Applications

Haiyang,

Ruomei,

Yan

et al. 2024

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Dissolution refers to the process in which solvent molecules and solute molecules attract and combine with each other. The extensive solubility data generated from the dissolution of various compounds under different conditions, is distributed across structured or semi‐structured formats in various media, such as text, web pages, tables, images, and databases. These data exhibit multi‐source and unstructured features, aligning with the typical 5 V characteristics of big data. A solubility big data technology system has emerged under the fusion of solubility data and big data technologies. However, the acquisition, fusion, storage, representation, and utilization of solubility big data are encountering new challenges. Knowledge Graphs, known as extensive systems for representing and applying knowledge, can effectively describe entities, concepts, and relations across diverse domains. The construction of solubility big data knowledge graph holds substantial value in the retrieval, analysis, utilization, and visualization of solubility knowledge. Throwing out a brick to attract a jade, this paper focuses on the solubility big data knowledge graph and, firstly, summarizes the architecture of solubility knowledge graph construction. Secondly, the key technologies such as knowledge extraction, knowledge fusion, and knowledge reasoning of solubility big data are emphasized, along with summarizing the common machine learning methods in knowledge graph construction. Furthermore, this paper explores application scenarios, such as knowledge question answering and recommender systems for solubility big data. Finally, it presents a prospective view of the shortcomings, challenges, and future directions related to the construction of solubility big data knowledge graph. This article proposes the research direction of solubility big data knowledge graph, which can provide technical references for constructing a solubility knowledge graph. At the same time, it serves as a comprehensive medium for describing data, resources, and their applications across diverse fields such as chemistry, materials, biology, energy, medicine, and so on. It further aids in knowledge retrieval and mining, analysis and utilization, and visualization across various disciplines.

show abstract

TSVFN: Two-Stage Visual Fusion Network for multimodal relation extraction

Cited by 25 publications

References 5 publications

MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation

MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation

Incorporating Object-Level Visual Context for Multimodal Fine-Grained Entity Typing

Knowledge Graph for Solubility Big Data: Construction and Applications

Contact Info

Product

Resources

About