KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction

Abstract

The ability of knowledge graphs to represent complex relationships at scale has led to their adoption for various needs including knowledge representation, question-answering, fraud detection, and recommendation systems. Knowledge graphs are often incomplete in the information they represent, necessitating the need for knowledge graph completion tasks, such as link and relation prediction. Pre-trained and fine-tuned language models have shown promise in these tasks although these models ignore the intrinsic information encoded in the knowledge graph, namely the entity and relation types. In this work, we propose the Knowledge Graph Language Model (KGLM) architecture, where we introduce a new entity/relation embedding layer that learns to differentiate distinctive entity and relation types, therefore allowing the model to learn the structure of the knowledge graph. In this work, we show that further pre-training the language models with this additional embedding layer using the triples extracted from the knowledge graph, followed by the standard fine-tuning phase sets a new state-of-the-art performance for the link prediction task on the benchmark datasets.

FoodAtlas: Automated Knowledge Extraction of Food and Chemicals from Literature

“…The link prediction models were also calibrated using isotonic regression to provide an interpretable probability score. Link prediction models are commonly evaluated using rank-based metrics like mean rank (MR), mean reciprocal rank (MRR), hits@1, hits@3, and hits@10 47 . However, our end goal was to generate hypotheses that were either true or false, and therefore, we decided to also evaluate using standard binary classification metrics like confusion matrix, precision, and recall.…”

SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design

“…Due to the recency of [37], we leave additional comparisons beyond our real GPT2 baseline to future work. Applying language models to knowledge graphs has been investigated in the general [91,30,93] and scientific domains [49,63]. They can be considered similar to our tests of BERT language models applied to a drug synergy hypergraph ( § 4.1).…”