Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder

Abstract

Embedding models for entities and relations are extremely useful for recovering missing facts in a knowledge base. Intuitively, a relation can be modeled by a matrix mapping entity vectors. However, relations reside on low dimension sub-manifolds in the parameter space of arbitrary matrices -for one reason, composition of two relations M 1 , M 2 may match a third M 3 (e.g. composition of relations currency of country and country of film usually matches currency of film budget), which imposes compositional constraints to be satisfied by the parameters (i.e. M 1 ·M 2 ≈ M 3 ). In this paper we investigate a dimension reduction technique by training relations jointly with an autoencoder, which is expected to better capture compositional constraints. We achieve state-of-the-art on Knowledge Base Completion tasks with strongly improved Mean Rank, and show that joint training with an autoencoder leads to interpretable sparse codings of relations, helps discovering compositional constraints and benefits from compositional training. Our source code is released at github.com/tianran/glimvec.

Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries

“…We train embeddings for 6 million Wikidata entities using feature-specific autoencoders to encode entity features such as names, aliases, description, entity types, and numeric attributes. This approach follows prior work on multimodal KB embeddings (Pezeshkpour et al, 2018) and learning of KB embeddings with autoencoders (Takahashi et al, 2018). Embedding training is detailed in Appendix D.…”

A survey of embedding models of entities and relationships for knowledge graph completion

“…Recent research has also shown that relation paths between entities in KGs provide richer context information and improve the performance of embedding models for KG completion (Luo et al, 2015;Liang and Forbus, 2015;García-Durán et al, 2015;Guu et al, 2015;Toutanova et al, 2016;Durán and Niepert, 2018;Takahashi et al, 2018;Chen et al, 2018). In particular, Luo et al (2015) constructed relation paths between entities and, viewing entities and relations in the path as pseudo-words, then applied Word2Vec (Mikolov et al, 2013) to produce pre-trained vectors for these pseudo-words.…”

Integrated Embedding Approach for Knowledge Base Completion with CNN

“…Most of subsequent models focus on integrating various role information brought by differ-ent relational attributes into entity representations for improvement. Lao et al [33][34] and Takahashi et al [59] verify the effectiveness of applying contextual information contained in relationship paths to KBC, demonstrating the value of relational features from another perspective. However, exquisitely designed models with stronger expressivity often come with higher computational costs.…”