Learning Semantically and Additively Compositional Distributional Representations

Tian, Ran; Okazaki, Naoaki; Inui, Kentaro

doi:10.18653/v1/p16-1121

Cited by 13 publications

(15 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to learn parameters u h , v t , M r of the score function, we follow Tian et al (2016) using a Noise Contrastive Estimation (NCE) (Gutmann and Hyvärinen, 2012) objective. For each path (or triple) h, r 1 / .…”

Section: Base Modelmentioning

confidence: 99%

Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder

Takahashi¹,

Tian

Inui

2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

Embedding models for entities and relations are extremely useful for recovering missing facts in a knowledge base. Intuitively, a relation can be modeled by a matrix mapping entity vectors. However, relations reside on low dimension sub-manifolds in the parameter space of arbitrary matrices -for one reason, composition of two relations M 1 , M 2 may match a third M 3 (e.g. composition of relations currency of country and country of film usually matches currency of film budget), which imposes compositional constraints to be satisfied by the parameters (i.e. M 1 ·M 2 ≈ M 3 ). In this paper we investigate a dimension reduction technique by training relations jointly with an autoencoder, which is expected to better capture compositional constraints. We achieve state-of-the-art on Knowledge Base Completion tasks with strongly improved Mean Rank, and show that joint training with an autoencoder leads to interpretable sparse codings of relations, helps discovering compositional constraints and benefits from compositional training. Our source code is released at github.com/tianran/glimvec.

show abstract

Section: Base Modelmentioning

confidence: 99%

Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder

Takahashi¹,

Tian

Inui

2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Since distributed representations play an important role in various NLP tasks, they are applied to semantics (Herbelot and Vecchi, 2015;Qiu et al, 2015;Woodsend and Lapata, 2015), with incorporating external information to them (Tian et al, 2016;Nguyen et al, 2016). In addition, finding interpretable regularities from the representations is often conducted through non-negative and sparse coding (Murphy et al, 2012;Faruqui et al, 2015;Luo et al, 2015;Kober et al, 2016), and regularization (Sun et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

Rotated Word Vector Representations and their Interpretability

Park¹,

Bak²,

Oh³

2017

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Vector representation of words improves performance in various NLP tasks, but the high-dimensional word vectors are very difficult to interpret. We apply several rotation algorithms to the vector representation of words to improve the interpretability. Unlike previous approaches that induce sparsity, the rotated vectors are interpretable while preserving the expressive performance of the original vectors. Furthermore, any pre-built word vector representation can be rotated for improved interpretability. We apply rotation to skipgrams and glove and compare the expressive power and interpretability with the original vectors and the sparse overcomplete vectors. The results show that the rotated vectors outperform the original and the sparse overcomplete vectors for interpretability and expressiveness tasks.

show abstract

“…In contrast, our proposal of the Near-far Context demonstrates that word order can be handled within an additive compositional framework, being parameterfree and with a proven bias bound. Recently, Tian et al (2016) further extended additive composition to realizing a formal semantics framework.…”

Section: Related Workmentioning

confidence: 99%

The mechanism of additive composition

2017

Self Cite

View full text Add to dashboard Cite

Additive composition (Foltz et al. in Discourse Process 15:285-307, 1998; Landauer and Dumais in Psychol Rev 104(2):211, 1997; Mitchell and Lapata in Cognit Sci 34(8):1388-1429, 2010) is a widely used method for computing meanings of phrases, which takes the average of vector representations of the constituent words. In this article, we prove an upper bound for the bias of additive composition, which is the first theoretical analysis on compositional frameworks from a machine learning point of view. The bound is written in terms of collocation strength; we prove that the more exclusively two successive words tend to occur together, the more accurate one can guarantee their additive composition as an approximation to the natural phrase vector. Our proof relies on properties of natural language data that are empirically verified, and can be theoretically derived from an assumption that the data is generated from a Hierarchical Pitman-Yor Process. The theory endorses additive composition as a reasonable operation for calculating meanings of phrases, and suggests ways to improve additive compositionality, including: transforming entries of distributional word vectors by a function that meets a specific condition, constructing a novel type of vector representations to make additive composition sensitive to word order, and utilizing singular value decomposition to train word vectors.

show abstract

Learning Semantically and Additively Compositional Distributional Representations

Cited by 13 publications

References 44 publications

Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder

Interpretable and Compositional Relation Learning by Joint Training with an Autoencoder

Rotated Word Vector Representations and their Interpretability

The mechanism of additive composition

Contact Info

Product

Resources

About