HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Chen, Sanxing; Liu, Xiaodong; Gao, Jianfeng; Jiao, Jian; Zhang, Ruofei; Ji, Yangfeng

doi:10.48550/arxiv.2008.12813

Cited by 3 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HittER is a hierarchical transformer model designed to learn the entity and relationship representations of knowledge graphs. This model demonstrates good performance in link prediction and question answering for datasets, such as FreebaseQA and WebQuestionSP [36]. KG-BERT uses a pretrained language model, such as BERT, to process the entity, relationship, and triple of the knowledge graph as a text sequence [37].…”

Section: Knowledge Graph Embedding and Completionmentioning

confidence: 99%

Recommendations for Responding to System Security Incidents Using Knowledge Graph Embedding

Kim,

Choi

2023

Electronics

View full text Add to dashboard Cite

Recently, security attacks occurring in edge computing environments have emerged as an important research topic in the field of cybersecurity. Edge computing is a distributed computing technology that expands the existing cloud computing architecture to introduce a new layer, the edge layer, between the cloud layer and the user terminal layer. Edge computing has the advantage of greatly improving the data processing speed and efficiency but, at the same time, is complex, and various new attacks occur frequently. Therefore, for improving the security of edge computing, effective and intelligent security strategies and policies must be established in consideration of a wide range of vulnerabilities. Intelligent security systems, which have recently been studied, provide a way to detect and respond to security threats by integrating the latest technologies, such as machine learning and big data analysis. Intelligent security technology can quickly recognize attack patterns or abnormal behaviors within a large amount of data and continuously respond to new threats through learning. In particular, knowledge-based technologies using ontology or knowledge graph technology play an important role in more deeply understanding the meaning and relationships between of security data and more effectively detecting and responding to complex threats. This study proposed a method for recommending strategies to respond to edge computing security incidents based on the automatic generation and embedding of security knowledge graphs. An EdgeSecurity–BERT model, utilizing the latest security vulnerability data from edge computing, was designed to extract entities and their relational information. Also, a security vulnerability assessment method was proposed to recommend strategies to respond to edge computing security incidents through knowledge graph embedding. In the experiment, the classification accuracy of security news data for common vulnerability and exposure data was approximately 86% on average. In addition, the EdgeSecurityKG applying the security vulnerability similarity improved the Hits@10 performance to identify the correct link, but the MR performance was degraded owing to the increased complexity. In complex areas, such as security, careful evaluation of the model’s performance and data selection are important. The EdgeSecurityKG applying the security vulnerability similarity provides an important advantage in understanding complex security vulnerability relationships.

show abstract

Section: Knowledge Graph Embedding and Completionmentioning

confidence: 99%

Recommendations for Responding to System Security Incidents Using Knowledge Graph Embedding

Kim,

Choi

2023

Electronics

View full text Add to dashboard Cite

show abstract

“…Recently, there has been a lot of interest in modifying this attention to further meet various desired specifications, e.g., to encode syntax trees (Strubell et al, 2018;Wang et al, 2019c), character-word lattice structures , as well as relative positions between words (Shaw et al, 2018;Wang et al, 2019a). There are also a few recent attempts that apply vanilla Transformer (Wang et al, 2019b) or hierarchical Transformer (Chen et al, 2020) to KGs, but mainly restricted to binary relations and deployed with conventional attention. This work, in contrast, deals with higher-arity relational data represented as heterogeneous graphs, and employs modified attention to encode graph structure and heterogeneity.…”

Section: Related Workmentioning

confidence: 99%

Link Prediction on N-ary Relational Facts: A Graph-based Approach

Wang

Lyu

et al. 2021

Preprint

View full text Add to dashboard Cite

Link prediction on knowledge graphs (KGs) is a key research topic. Previous work mainly focused on binary relations, paying less attention to higher-arity relations although they are ubiquitous in real-world KGs. This paper considers link prediction upon n-ary relational facts and proposes a graph-based approach to this task. The key to our approach is to represent the nary structure of a fact as a small heterogeneous graph, and model this graph with edge-biased fully-connected attention. The fully-connected attention captures universal inter-vertex interactions, while with edge-aware attentive biases to particularly encode the graph structure and its heterogeneity. In this fashion, our approach fully models global and local dependencies in each n-ary fact, and hence can more effectively capture associations therein. Extensive evaluation verifies the effectiveness and superiority of our approach. It performs substantially and consistently better than current state-of-the-art across a variety of n-ary relational benchmarks. Our code is publicly available. 1

show abstract

“…Though some efforts have been made to incorporate neighborhood into the KGC algorithms (Chen et al, 2020), our proposed pipeline inherits the benefits of generative LLMs compared to KGE approaches: (1) scalability and size of the model, (2) applicability to both transductive and inductive KGs due to an ability to generalize unseen entities, (3) no need to rank all possible candidate triplets due to direct generation of tail entity. Also, a concurrent work by the KGT5 authors, KGT5context (Kochsiek et al, 2023), proposed a similar idea of integrating node neighborhood in the context of the generative LM model, supporting the main concerns and results of our study.…”

Section: Introductionmentioning

confidence: 99%

Better Together: Enhancing Generative Knowledge Graph Completion with Language Models and Neighborhood Information

Chepurova,

Bulatov,

Kuratov

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Real-world Knowledge Graphs (KGs) often suffer from incompleteness, which limits their potential performance. Knowledge Graph Completion (KGC) techniques aim to address this issue. However, traditional KGC methods are computationally intensive and impractical for large-scale KGs, necessitating the learning of dense node embeddings and computing pairwise distances. Generative transformer-based language models (e.g., T5 and recent KGT5) offer a promising solution as they can predict the tail nodes directly. In this study, we propose to include node neighborhoods as additional information to improve KGC methods based on language models. We examine the effects of this imputation and show that, on both inductive and transductive Wikidata subsets, our method outperforms KGT5 and conventional KGC approaches. We also provide an extensive analysis of the impact of neighborhood on model prediction and show its importance. Furthermore, we point the way to significantly improve KGC through more effective neighborhood selection.

show abstract

HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Cited by 3 publications

References 27 publications

Recommendations for Responding to System Security Incidents Using Knowledge Graph Embedding

Recommendations for Responding to System Security Incidents Using Knowledge Graph Embedding

Link Prediction on N-ary Relational Facts: A Graph-based Approach

Better Together: Enhancing Generative Knowledge Graph Completion with Language Models and Neighborhood Information

Contact Info

Product

Resources

About