Bringing Light Into the Dark: A Large-Scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework

Abstract

The heterogeneity in recently published knowledge graph embedding models' implementations, training, and evaluation has made fair and thorough comparisons difficult. To assess the reproducibility of previously published results, we re-implemented and evaluated 21 models in the PyKEEN software package. In this paper, we outline which results could be reproduced with their reported hyper-parameters, which could only be reproduced with alternate hyper-parameters, and which could not be reproduced at all, as well as provide insight as to why this might be the case.We then performed a large-scale benchmarking on four datasets with several thousands of experiments and 24,804 GPU hours of computation time. We present insights gained as to best practices, best configurations for each model, and where improvements could be made over previously published best configurations. Our results highlight that the combination of model architecture, training approach, loss function, and the explicit modeling of inverse relations is crucial for a model's performance and is not only determined by its architecture. We provide evidence that several architectures can obtain results competitive to the state of the art when configured carefully. We have made all code, experimental configurations, results, and analyses available at https://github. com/pykeen/pykeen and https://github.com/pykeen/benchmarking.

Distributed non-negative RESCAL with Automatic Model Selection for Exascale Data

“…Tensor factorization and tensor networks [12] are wellknown factorization techniques used for knowledge graph embedding with semantic matching interactions [13]. In a tensor representation, each triplet of the knowledge graph is represented by the value, X ijk of a three dimensional tensor, X ∈ R n×n×m the triplet (i, j, k) exists, or can be an arbitrary non-negative value that measures the strength of the k-relation between the iand j-entities.…”

Query Embedding on Hyper-relational Knowledge Graphs

“…On the other hand, QE bypasses the need for a database or query engine and performs reasoning directly in a latent space by computing a similarity score between the query representation and entity representations 1 . A query representation is obtained by processing its equivalent logical formula where joins become intersections (∧), and variables are existentially quantified (∃).…”

“…Instead, CQD decomposes a query in a sequence of reasoning steps and performs a beam search in a latent space of KG embeddings models pre-trained on a simple 1-hop link prediction task. A particular novelty of this approach is that no end-to-end training on complex queries is required, and any trained embedding model of the existing abundance [1,16] can be employed as is.…”

“…Besides that, we propose a method to answer such queries and demonstrate in our experiments that qualifiers improve query answering on a diverse set of query patterns. 1 Existing QE approaches operate only on the entity level and cannot have relations as variables Preprint. Under review.…”

BioKEEN: a library for learning and evaluating biological knowledge graph embeddings

“…Existing software packages that provide implementations for different KGEMs usually lack entire composability: model architectures (or interaction models), training approaches, loss functions, and the usage of explicit inverse relation cannot arbitrarily be combined. The full composability of KGEMs is fundamental for assessing the performance of KGEMs because it allows assessing single components individually on the model's performance instead of attributing a performance increase solely to the model architecture, which is misleading (Ruffinelli et al, 2020;Ali et al, 2020). Besides, often only limited functionalities are provided, e.g., a small number of KGEMs are supported, or functionalities such as HPO are missing.…”