Relational Message Passing for Knowledge Graph Completion

Wang, Hongwei; Ren, H.; Leskovec, Jure

doi:10.1145/3447548.3467247

Cited by 91 publications

(55 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, for link prediction, GNN models, such as GCN (Kipf & Welling, 2017), GAE (Kipf & Welling, 2016) may perform even worse than some simple heuristics such as common neighbors and Adamic Adar (Liben-Nowell & Kleinberg, 2007) (see the performance comparison over the networks Collab and PPA in Open Graph Benchmark (OGB) (Hu et al, 2020)). Similar issues widely appear in node-set-based tasks such as network motif prediction (Liu et al, 2022;Besta et al, 2021), motif counting (Chen et al, 2020), relation prediction (Wang et al, 2021a;Teru et al, 2020) and temporal interaction prediction (Wang et al, 2021b), which posts a big concern for applying GNNs to these relevant real-world applications.…”

Section: Introductionmentioning

confidence: 81%

Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks

Wang¹,

Yin²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Graph neural networks (GNN) have shown great advantages in many graph-based learning tasks but often fail to predict accurately for a task based on sets of nodes such as link/motif prediction and so on. Many works have recently proposed to address this problem by using random node features or node distance features. However, they suffer from either slow convergence, inaccurate prediction or high complexity. In this work, we revisit GNNs that allow using positional features of nodes given by positional encoding (PE) techniques such as Laplacian Eigenmap, Deepwalk, etc.. GNNs with PE often get criticized because they are not generalizable to unseen graphs (inductive) or stable. Here, we study these issues in a principled way and propose a provable solution, a class of GNN layers termed PEG with rigorous mathematical analysis. PEG uses separate channels to update the original node features and positional features. PEG imposes permutation equivariance w.r.t. the original node features and rotation equivariance w.r.t. the positional features simultaneously. Extensive link prediction experiments over 8 real-world networks demonstrate the advantages of PEG in generalization and scalability. 1

show abstract

Section: Introductionmentioning

confidence: 81%

Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks

Wang¹,

Yin²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This corresponds to our MLM task, where masked tokens in Segment A can be predicted using Segment B (a linked document on the graph), and vice versa. In link prediction (Bordes et al, 2013;Wang et al, 2021a), the task is to predict the existence or type of an edge between two nodes. This corresponds to our DRP task, where we predict if the given pair of text segments are linked (edge), contiguous (self-loop edge), or random (no edge).…”

Section: Pretraining Tasksmentioning

confidence: 99%

LinkBERT: Pretraining Language Models with Document Links

Yasunaga¹,

Leskovec²,

Liang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not capture dependencies or knowledge that span across documents. In this work, we propose LinkBERT, an LM pretraining method that leverages links between documents, e.g., hyperlinks. Given a text corpus, we view it as a graph of documents and create LM inputs by placing linked documents in the same context. We then pretrain the LM with two joint self-supervised objectives: masked language modeling and our new proposal, document relation prediction. We show that LinkBERT outperforms BERT on various downstream tasks across two domains: the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links). LinkBERT is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT sets new states of the art on various BioNLP tasks (+7% on BioASQ and USMLE). We release our pretrained models, LinkBERT and BioLinkBERT, as well as code and data. 1 * Equal senior authorship. 1 Available at https://github.com/michiyasunaga/ LinkBERT.

show abstract

“…In describing the post-training process, we have assumed that the original model only leverages individual facts, which is by far the dominant approach in literature. A few recent methods can also exploit contextual information, such as paths [18,55], temporal details [29], or types [59]. While our current implementation focuses on fact-based models, the formulation of Kelpie can indeed be applied to these contextual models too: the Pre-Filter and the Explanation Builder, which are model-independent, would just work as usual, and the Relevance Engine could easily include contextual information in its post-training processes.…”

Section: Relevance Enginementioning

confidence: 99%

Explaining Link Prediction Systems based on Knowledge Graph Embeddings

Rossi

Firmani

Merialdo

et al. 2022

Proceedings of the 2022 International Conference on Management of Data

View full text Add to dashboard Cite

Link Prediction (LP) aims at tackling Knowledge Graph incompleteness by inferring new, missing facts from the already known ones. The rise of novel Machine Learning techniques has led researchers to develop LP models that represent Knowledge Graph elements as vectors in an embedding space. These models can outperform traditional approaches and they can be employed in multiple downstream tasks; nonetheless, they tend to be opaque, and are mostly regarded as black boxes. Their lack of interpretability limits our understanding of their inner mechanisms, and undermines the trust that users can place in them. In this paper, we propose the novel Kelpie explainability framework. Kelpie can be applied to any embedding-based LP models independently from their architecture, and it explains predictions by identifying the combinations of training facts that have enabled them. Kelpie can extract two complementary types of explanations, that we dub necessary and sufficient. We describe in detail both the structure and the implementation details of Kelpie, and thoroughly analyze its performance through extensive experiments. Our results show that Kelpie significantly outperforms baselines across almost all scenarios.

show abstract

Relational Message Passing for Knowledge Graph Completion

Cited by 91 publications

References 18 publications

Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks

Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks

LinkBERT: Pretraining Language Models with Document Links

Explaining Link Prediction Systems based on Knowledge Graph Embeddings

Contact Info

Product

Resources

About