Interpretable and Fair Comparison of Link Prediction or Entity Alignment Methods

Abstract

In this work, we take a closer look at the evaluation of two families of methods for enriching information from knowledge graphs: Link Prediction and Entity Alignment. In the current experimental setting, multiple different scores are employed to assess different aspects of model performance. We analyze the informative value of these evaluation measures and identify several shortcomings. In particular, we demonstrate that all existing scores can hardly be used to compare results across different datasets. Moreover, this problem may also arise when comparing different train/test splits for the same dataset. We show that this leads to various problems in the interpretation of results, which may support misleading conclusions. Therefore, we propose a different evaluation and demonstrate empirically how this helps for fair, comparable and interpretable assessment of model performance.

Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework

“…A further interesting study with a different focus is the work of Rossi et al [9] in which they investigated the effect of the structural properties of KGs on models' performances instead of the combinations of different model architectures, training approaches, and loss functions. It should be considered that in several studies, different implementations of KGEMs have been combined, which can be critical for a fair and transparent comparison [10], [4], [11].…”

“…In Section III, we present our definition of a KGEM and review the KGEMs that we investigated in our studies. In Section IV, we describe and discuss established evaluation metrics as well as a recently proposed one [10]. In Section V, we introduce the benchmark datasets on which we conducted our experiments.…”

Query Embedding on Hyper-relational Knowledge Graphs

“…We provide our chosen hyper-parameters after performing hyper-parameter optimisation in table 3 and detailed results including standard deviation across five runs with different random seeds in Tables 4 and 5. We report Hits@k for k = 1, 3, 10, mean reciprocal rank (MRR), and adjusted mean rank index (AMRI) [4]. For all metrics, larger values indicate better performance.…”

“…Hits@1 (%) StarQE Table 5: Full results for the generalization experiments, including standard deviation across five runs with different random seeds. We report Hits@k for k = 1, 3, 10, mean reciprocal rank (MRR), and adjusted mean rank index (AMRI) [4]. For all metrics, larger values indicate better performance.…”

“…We compute filtered ranks [5], i.e., while computing the rank of a correct entity, we ignore the scores of other correct entities. We resolve exactly equal scores using the realistic rank [4], i.e., all entities with equal score obtain the rank of the average of the first and last position.…”