From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph

Abstract

Knowledge graphs are usually constructed to describe the various concepts that exist in real world as well as the relationships between them. There are many knowledge graphs in specific fields, but they usually pay more attention on text or structured data, ignoring the image vision information, and cannot play an adequate role in the emerging visualization applications. Aiming at this issue, we design a method that integrates image vision information and text information derived from Wikimedia Commons to construct a domain-specific multi-modal knowledge graph, taking the metallic materials domain as an example to illustrate the method. The text description of each image is regarded as its context semantic to acquire the image's context semantic labels based on the DBpedia resource. Furthermore, we adopt deep neural network model instead of simple visual descriptors to acquire the image's visual semantic labels using the concepts from WordNet. In order to fuse the visual semantic labels and context semantic labels, a path-based concept extension and fusion strategy is proposed based on the conceptual hierarchies of WordNet and DBpedia to obtain the effective extension concepts as well as the links between them, increasing the scale of the knowledge graph and enhancing the correlation between images. The experimental results show that the maximum extension level has a significant impact on the quality of the generated domain knowledge graph, and the best extension level number is respectively determined for both DBpedia and WordNet. In addition, the results of this paper are compared with IMGpedia to further show the effectiveness of the proposed method. INDEX TERMS Knowledge graph, information fusion, multi-modal knowledge, metallic materials knowledge. IV. CONSTRUCTION METHOD OF DOMAIN-SPECIFIC MULTI-MODAL KNOWLEDGE GRAPH A. APPROACH OVERVIEW

Multi-Modal Knowledge Graph Construction and Application: A Survey

“…Entity grounding methods make it possi- Compared to the encyclopedias-based approach, search engine based approach is better in coverage but worse in quality. The two approaches are often used together since in most cases the knowledge acquired by these two approaches complement with each other [25], [100]. For example, the coverage of MMKG harvested from Wikipedia can be improved by collecting more images for each entity from search engines [25] or mapping each image to all the entities it contains to expand the number of entities' images [100].…”

A survey on the development status and application prospects of knowledge graph in smart grids

“…To extract valuable knowledge from massive heterogeneous power data depending only on the domain knowledge of traditional expert systems is neither efficient nor sufficient. Hence, it is essential and beneficial for all the stakeholders in their power sector to develop more efficient models to better make use of multi-modal electricity knowledge and unlock underlying information and relations by the cross-fertilization of the multi-source power data [181].…”

Graph Based Analytics and Review of Knowledge Graph Field by Using Two Decade Data- Finding Paradigms and Opportunities

“…Knowledge graphs complement machine learning techniques to reduce the need for large, titled datasets, facilitate transfer learning, and explain and encode domain, task, and application knowledge that would be costly to learn from data alone 8,9,10) . A knowledge graph organizes and integrates data according to an ontology 13,15) , which is called the schema of the knowledge graph, and applies a reasoner to derive new knowledge.…”