AYNEC: All You Need for Evaluating Completion Techniques in Knowledge Graphs

Ayala, Daniel; Borrego, Agustín; Hernández, Inma; Rivero, Carlos R.; Ruiz, David

doi:10.1007/978-3-030-21348-0_26

Cited by 8 publications

(9 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bonner et al [3] 2017 Topological structure Model GEM [7] 2018 Clas, Clu, LP, Net Comp, Vis Model Rulinda et al [16] [11] 2019 LP Model AYNEC [2] 2019 LP -Bogumil et al [9] 2019 Clu Model GEval 2019 Clas, Clu, DocSim, EntRel, Reg, SemAn…”

Section: Year Tasks Embedding Techniquementioning

confidence: 99%

See 1 more Smart Citation

GEval: A Modular and Extensible Evaluation Framework for Graph Embedding Techniques

et al. 2020

View full text Add to dashboard Cite

While RDF data are graph shaped by nature, most traditional Machine Learning (ML) algorithms expect data in a vector form. To transform graph elements to vectors, several graph embedding approaches have been proposed. Comparing these approaches is interesting for 1) developers of new embedding techniques to verify in which cases their proposal outperforms the state-of-art and 2) consumers of these techniques in choosing the best approach according to the task(s) the vectors will be used for. The comparison could be delayed (and made difficult) by the choice of tasks, the design of the evaluation, the selection of models, parameters, and needed datasets. We propose GEval, an evaluation framework to simplify the evaluation and the comparison of graph embedding techniques. The covered tasks range from ML tasks (Classification, Regression, Clustering), semantic tasks (entity relatedness, document similarity) to semantic analogies. However, GEval is designed to be (easily) extensible. In this article, we will describe the design and development of the proposed framework by detailing its overall structure, the already implemented tasks, and how to extend it. In conclusion, to demonstrate its operating approach, we consider the parameter tuning of the KGloVe algorithm as a use case.

show abstract

Section: Year Tasks Embedding Techniquementioning

confidence: 99%

“…Also AYNEC [2] focuses on the link prediction task. It provides some incomplete graphs as a training set.…”

Section: Vectorsmentioning

confidence: 99%

GEval: A Modular and Extensible Evaluation Framework for Graph Embedding Techniques

et al. 2020

View full text Add to dashboard Cite

show abstract

“…We evaluated CHAI using four different Knowledge Graphs that are openly available and commonly used for the task of KG completion: FB13 [26], WN18 [6] (which are subsets of Freebase [3] and Wordnet [20], respectively), a subset of NELL introduced by Gardner and Mitchell [11], and EPSRC 1 , which contains information about the grants provided by the Engineering and Physical Sciences Research Council of the United Kingdom. All of these datasets were obtained from the publicly available AYNEC-DataGen tool [1], and an overview of their metadata can be found in Table 1. We used CHAI to generate rules for every relation in every dataset, except for the NELL dataset, in which we focused on the same subset of 10 relations as Gardner and Mitchell [11] due to the high number of total relations.…”

Section: Setup and Datasetsmentioning

confidence: 99%

Generating Rules to Filter Candidate Triples for their Correctness Checking by Knowledge Graph Completion Techniques

Borrego

Ayala

Hernández

et al. 2019

Proceedings of the 10th International Conference on Knowledge Capture

Self Cite

View full text Add to dashboard Cite

Knowledge Graphs (KGs) contain large amounts of structured information. Due to their inherent incompleteness, a process known as KG completion is often carried out to find the missing triples in a KG, usually by training a fact checking model that is able to discern between correct and incorrect knowledge. After the fact checking model has been trained and evaluated, it has to be applied to a set of candidate triples, and those that are considered correct are added to the KG as new knowledge. However, this process needs a set of candidate triples of a reasonable size that represents possible new knowledge, in order to be evaluated by the fact checking task and, if considered to be correct, added to the KG, enriching it. Current approaches for selecting candidate triples for their correctness checking either use the full set possible missing candidate triples (and thus provide no filtering) or apply very basic rules to filter out unlikely candidates, which may have a negative effect on the completion performance as very few candidate triples are filtered out. In this paper we present CHAI, a method for producing more complex rules that are able to filter candidate triples by combining a set of criteria to optimize a fitness function. Our experiments show that CHAI is able to generate rules that, when applied, yield smaller candidate sets than similar proposals while still including promising candidate triples. •

show abstract

“…To accomplish their goal, these algorithms typically train a model that takes candidate triples as input and outputs whether they are correct (positive) and should be added to the graph, or incorrect (negative) and should be discarded [6]. The knowledge graph at hand must thus be divided into a training, a validation (optionally) and a test splits (subgraphs) [2].…”

Section: Introductionmentioning

confidence: 99%

“…As a result, several strategies have been proposed to generate negative counterparts [14][15][16][17]. The accuracy of completion models is known to be highly dependent on the negative triples used during training [2,5,11,15,27]. However, as far as we know, a thorough analysis on how accuracy is impacted by the different negative triple generation strategies is still missing.…”

Section: Introductionmentioning

confidence: 99%

The Impact of Negative Triple Generation Strategies and Anomalies on Knowledge Graph Completion

Bansal

Tiwari

Rivero

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

Self Cite

View full text Add to dashboard Cite

Even though knowledge graphs have proven very useful for several tasks, they are marked by incompleteness. Completion algorithms aim to extend knowledge graphs by predicting missing (subject, predicate, object) triples, usually by training a model to discern between correct (positive) and incorrect (negative) triples. However, under the open-world assumption in which a missing triple is not negative but unknown, negative triple generation is challenging. Although negative triples are known to drive the accuracy of completion models, its impact has not been thoroughly examined yet. To evaluate accuracy, test triples are considered positive and negative triples are derived from them. The evaluation protocol is thus impacted by the generation of negative triples, which remains to be analyzed. Another issue is that the knowledge graphs available for evaluation contain anomalies like severe redundancy, and it is unclear how anomalies affect the accuracy of completion models. In this paper, we analyze the impact of negative triple generation during both training and testing on translation-based completion models. We examine four negative triple generation strategies, which are also used to evaluate the models when anomalies in the test split are included and discarded. In addition to previously-studied anomalies like near-same predicates, we include another anomaly: knowledge present in the test that is missing from the training split. Our main conclusion is that the most common strategy for negative triple generation (local-closed world assumption) can be mimicked by a combination of a naïve and a immediate neighborhood strategies. This result suggests that completion models can be learned independently for certain subgraphs, which would render completion models useful in the context of knowledge graph evolution. Although anomalies are considered harmful since they artificially increase the accuracy of completion models, our results show otherwise for certain knowledge graphs, which calls for further research efforts. CCS CONCEPTS • Computing methodologies → Semantic networks; Supervised learning; • Information systems → Inconsistent data.

show abstract

AYNEC: All You Need for Evaluating Completion Techniques in Knowledge Graphs

Cited by 8 publications

References 17 publications

GEval: A Modular and Extensible Evaluation Framework for Graph Embedding Techniques

GEval: A Modular and Extensible Evaluation Framework for Graph Embedding Techniques

Generating Rules to Filter Candidate Triples for their Correctness Checking by Knowledge Graph Completion Techniques

The Impact of Negative Triple Generation Strategies and Anomalies on Knowledge Graph Completion

Contact Info

Product

Resources

About