A Unified Framework for Rank-based Evaluation Metrics for Link Prediction in Knowledge Graphs

Hoyt, Charles Tapley; Berrendorf, Max; Galkin, Mikhail; Tresp, Volker; Gyori, Benjamin M.

doi:10.48550/arxiv.2203.07544

Cited by 4 publications

(8 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…KGEM performance is almost exclusively assessed using the following rank-based metrics: Hits@K, Mean Rank (MR), and Mean Reciprocal Rank (MRR) [19]. Disagreements exist as to how and when these metrics can be used and compared properly.…”

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

“…Because this metric does not use any threshold K compared to Hits@K, it is less sensitive to outliers. In addition, it is often used for performing early stopping and for tracking the best epoch during training [5,19].…”

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

“…In addition, the aforementioned rank-based metrics have intrinsic and theoretical flaws, as pointed out in several works [5,19,64]. For example, Hits@K does not take into account triples whose rank is larger than K. As such, a model scoring the ground-truth in position K + 1 would be considered equally good as another model scoring the ground-truth in position K + d with d…”

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

“…1. It follows that Hits@K is not a suitable metric for drawing comparisons between models [19]. MR alleviates this concern as it does not consider any threshold K. Therefore, MR allows to compare KGEM performance on the same dataset.…”

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

“…The performance of KGEM for LP is evaluated using rank-based metrics such as Hits@K, Mean Rank (MR), and Mean Reciprocal Rank (MRR) that assess whether ground-truth entities are indeed given higher scores [54,72]. However, various works recently raised some caveats about such metrics [5,19,64]. For instance, they are not wellsuited for drawing comparisons across datasets [5].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Sem@K: Is my knowledge graph embedding model semantic-aware?

Hubert,

Monnin,

Brun

et al. 2023

View full text Add to dashboard Cite

Using knowledge graph embedding models (KGEMs) is a popular approach for predicting links in knowledge graphs (KGs). Traditionally, the performance of KGEMs for link prediction is assessed using rank-based metrics, which evaluate their ability to give high scores to ground-truth entities. However, the literature claims that the KGEM evaluation procedure would benefit from adding supplementary dimensions to assess. That is why, in this paper, we extend our previously introduced metric Sem@K that measures the capability of models to predict valid entities w.r.t. domain and range constraints. In particular, we consider a broad range of KGs and take their respective characteristics into account to propose different versions of Sem@K. We also perform an extensive study to qualify the abilities of KGEMs as measured by our metric. Our experiments show that Sem@K provides a new perspective on KGEM quality. Its joint analysis with rank-based metrics offers different conclusions on the predictive power of models. Regarding Sem@K, some KGEMs are inherently better than others, but this semantic superiority is not indicative of their performance w.r.t. rank-based metrics. In this work, we generalize conclusions about the relative performance of KGEMs w.r.t. rank-based and semantic-oriented metrics at the level of families of models. The joint analysis of the aforementioned metrics gives more insight into the peculiarities of each model. This work paves the way for a more comprehensive evaluation of KGEM adequacy for specific downstream tasks.

show abstract

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

Section: Evaluating Kgem Performance For Link Predictionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sem@K: Is my knowledge graph embedding model semantic-aware?

Hubert,

Monnin,

Brun

et al. 2023

View full text Add to dashboard Cite

show abstract

Cluster Robust Inference for Embedding-Based Knowledge Graph Completion

Schramm

Niklas

Schmid

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Semantic-Enhanced Knowledge Graph Completion

Yuan,

Chen,

Wang

et al. 2024

Mathematics

View full text Add to dashboard Cite

Knowledge graphs (KGs) serve as structured representations of knowledge, comprising entities and relations. KGs are inherently incomplete, sparse, and have a strong need for completion. Although many knowledge graph embedding models have been designed for knowledge graph completion, they predominantly focus on capturing observable correlations between entities. Due to the sparsity of KGs, potential semantic correlations are challenging to capture. To tackle this problem, we propose a model entitled semantic-enhanced knowledge graph completion (SE-KGC). SE-KGC effectively addresses the issue by incorporating predefined semantic patterns, enabling the capture of semantic correlations between entities and enhancing features for representation learning. To implement this approach, we employ a multi-relational graph convolution network encoder, which effectively encodes the KG. Subsequently, we utilize a scoring decoder to evaluate triplets. Experimental results demonstrate that our SE-KGC model outperforms other state-of-the-art methods in link-prediction tasks across three datasets. Specifically, compared to the baselines, SE-KGC achieved improvements of 11.7%, 1.05%, and 2.30% in terms of MRR on these three datasets. Furthermore, we present a comprehensive analysis of the contributions of different semantic patterns, and find that entities with higher connectivity play a pivotal role in effectively capturing and characterizing semantic information.

show abstract

A Unified Framework for Rank-based Evaluation Metrics for Link Prediction in Knowledge Graphs

Cited by 4 publications

References 20 publications

Sem@K: Is my knowledge graph embedding model semantic-aware?

Sem@K: Is my knowledge graph embedding model semantic-aware?

Cluster Robust Inference for Embedding-Based Knowledge Graph Completion

Semantic-Enhanced Knowledge Graph Completion

Contact Info

Product

Resources

About