On the Ambiguity of Rank-Based Evaluation of Entity Alignment or Link Prediction Methods

Berrendorf, Max; Faerman, Evgeniy; Vermue, Laurent; Tresp, Volker

doi:10.48550/arxiv.2002.06914

Cited by 8 publications

(31 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We modify our code to construct queries in a similar fashion, though we still use random-break ranking, The grouping of results by embedding based methods, rule based methods, and those obtained by our code is as before. MINERVA and ConvE use nondeterministic [Berrendorf et al, 2021] ranking, as they sort scores before ranking, but NTP does not. The NTP code refers to the ranking method in ComplEx and says that "we calculate the rank from only those corrupted triples that have a higher score" -i.e., they use optimistic ranking.…”

Section: Discussionmentioning

confidence: 99%

“…The results for the other embedding-based methods were taken from either the original papers or from Dettmers et al [2018] or Qu et al [2021]. These codes do not implement random break ranking for equal scores, and some instead use nondeterministic ranking [Berrendorf et al, 2021], i.e., they sort entity scores before ranking. This approach is fine in the absence of equal scores for multiple entities, which we believe is the case for embedding based methods.…”

Section: Methodsmentioning

confidence: 99%

“…a. Entity Ranking: There are different ways [Berrendorf et al, 2021, Sun et al, 2020 of ranking entities e that form candidate solutions to queries (t, r, ?) and (?, r, h) constructed from a test set fact (t, r, h), and the choice of the ranking method may have a significant effect on the final MRR, especially for rule-based methods, see Table 7.…”

Section: A Appendixmentioning

confidence: 99%

“…and n entities receive the same score. Some ranking methods listed in Berrendorf et al [2021] are: h receives a rank of m + 1 (optimistic), i.e., the best among all equally-scored entities, or m + n (pessimistic), or k…”

Section: A Appendixmentioning

confidence: 99%

“…where k is a random number between m + 1 and m + n (random [Sun et al, 2020]). It is also observed in Berrendorf et al [2021] that numerous codes give a nondeterministic rank between m + 1 and m + n based on the position of the correct entity in a sorted order of equal score entities. Other options are a rank of m + (n − m)/2 (midpoint) [Qu et al, 2021] or random…”

Section: A Appendixmentioning

confidence: 99%

See 4 more Smart Citations

Rule Induction in Knowledge Graphs Using Linear Programming

Dash¹,

Gonçalves²

2021

Preprint

View full text Add to dashboard Cite

Knowledge graph (KG) completion is a well-studied problem in AI. Rule-based methods and embedding-based methods form two of the solution techniques. Rule-based methods learn first-order logic rules that capture existing facts in an input graph and then use these rules for reasoning about missing facts. A major drawback of such methods is the lack of scalability to large datasets. In this paper, we present a simple linear programming (LP) model to choose rules from a list of candidate rules and assign weights to them. For smaller KGs, we use simple heuristics to create the candidate list. For larger KGs, we start with a small initial candidate list, and then use standard column generation ideas to add more rules in order to improve the LP model objective value. To foster interpretability and generalizability, we limit the complexity of the set of chosen rules via explicit constraints, and tune the complexity hyperparameter for individual datasets. We show that our method can obtain state-of-the-art results for three out of four widely used KG datasets, while taking significantly less computing time than other popular rule learners including some based on neuro-symbolic methods. The improved scalability of our method allows us to tackle large datasets such as YAGO3-10.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: A Appendixmentioning

confidence: 99%

Section: A Appendixmentioning

confidence: 99%

Section: A Appendixmentioning

confidence: 99%

See 3 more Smart Citations

Rule Induction in Knowledge Graphs Using Linear Programming

Dash¹,

Gonçalves²

2021

Preprint

View full text Add to dashboard Cite

show abstract

Marrying Query Rewriting and Knowledge Graph Embeddings

Imenes,

Guimarães,

Ozaki

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

An Ensemble Learning Approach to perform Link Prediction on Large Scale Biomedical Knowledge Graphs for Drug Repurposing and Discovery

Prabhakar¹,

Vu²,

Crawford³

et al. 2023

Preprint

View full text Add to dashboard Cite

Generating knowledge graph embeddings (KGEs) to represent entities (nodes) and relations (edges) in large scale knowledge graph datasets has been a challenging problem in representation learning. This is primarily because the embeddings / vector representations that are required to encode the full scope of data in a large heterogeneous graph needs to have a high dimensionality. The orientation of a large number of vectors requires a lot of space which is achieved by projecting the embeddings to higher dimensions. This is not a scalable solution especially when we expect the knowledge graph to grow in size in order to incorporate more data. Any efforts to constrain the embeddings to lower number of dimensions could be problematic as insufficient space to spatially orient the large number of embeddings / vector representations within limited number of dimensions could lead to poor inferencing on downstream tasks such as link prediction which leverage these embeddings to predict the likelihood of existence of a link between two or more entities in a knowledge graph. This is especially the case with large biomedical knowledge graphs which relate several diverse entities such as genes, diseases, signaling pathways, biological functions etc. that are clinically relevant for the application of KGs to drug discovery. The size of the biomedical knowledge graphs are therefore much larger compared to typical benchmark knowledge graph datasets. This poses a huge challenge in generating embeddings / vector representations of good quality to represent the latent semantic structure of the graph. Attempts to circumvent this challenge by increasing the dimensionality of the embeddings often render hardware limitations as generating high dimensional embeddings is computationally expensive and often times infeasible. To practically deal with representing the latent structure of such large scale knowledge graphs (KGs), our work proposes an ensemble learning model in which the full knowledge graph is sampled into several smaller subgraphs and KGE models generate embeddings for each individual subgraph. The results of link prediction from the KGE models trained on each subgraph are then aggregated to generate a consolidated set of link predictions across the full knowledge graph. The experimental results demonstrated significant improvement in rank-based evaluation metrics on task specific link predictions as well as general link predictions on four open-sourced biomedical knowledge graph datasets.

show abstract

On the Ambiguity of Rank-Based Evaluation of Entity Alignment or Link Prediction Methods

Cited by 8 publications

References 0 publications

Rule Induction in Knowledge Graphs Using Linear Programming

Rule Induction in Knowledge Graphs Using Linear Programming

Marrying Query Rewriting and Knowledge Graph Embeddings

An Ensemble Learning Approach to perform Link Prediction on Large Scale Biomedical Knowledge Graphs for Drug Repurposing and Discovery

Contact Info

Product

Resources

About