A Latent Variable Model Approach to PMI-based Word Embeddings

Arora, Sanjeev; Li, Yuanzhi; Liang, Yingyu; Ma, Tengyu; Risteski, Andrej

doi:10.48550/arxiv.1502.03520

Cited by 13 publications

(35 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that c has a uniform distribution over the unit sphere. In this case, from Lemma A.5 in (Arora et al, 2015), (53) holds approximately.…”

Section: Learning With Multiple Negative Triplesmentioning

confidence: 89%

“…If c was distributed as N (0, 1 d I), this would be a simple tail bound. However, as c is distributed uniformly on the sphere, this requires special care, and the claim follows by applying the tail bound for the spherical distribution given by Lemma A.1 in (Arora et al, 2015) instead. Finally, applying Corollary A.3 in (Arora et al, 2015), we have:…”

Section: B Proof Of Relwalk Theoremmentioning

confidence: 99%

“…Let us denote by z the random variable 2h R 1 c. Moreover, let r(z) = E c |z [1 F ], which is a function of z between [0, 1]. We wish to upper bound E c [exp(z)r(z)].The worst-case r(z) can be quantified using a continuous version of Abel's inequality (proved as Lemma A.4 in(Arora et al, 2015)), we can upper bound E c [exp(z)r(z)] as follows:E c [exp(z)r(z)] ≤ E exp(z)1 [t,+∞] (z) (39) where t satisfies that E c [1 [t,+∞] (z)] = Pr[z ≥ t] = E c [r(z)] ≤ exp(−Ω(log 2 n)).…”

mentioning

confidence: 99%

See 2 more Smart Citations

RelWalk - A Latent Variable Model Approach to Knowledge Graph Embedding

Bollegala

Hakami

Yoshida

et al. 2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

View full text Add to dashboard Cite

Embedding entities and relations of a knowledge graph in a low-dimensional space has shown impressive performance in predicting missing links between entities. Although progresses have been achieved, existing methods are heuristically motivated and theoretical understanding of such embeddings is comparatively underdeveloped. This paper extends the random walk model (Arora et al., 2016a) of word embeddings to Knowledge Graph Embeddings (KGEs) to derive a scoring function that evaluates the strength of a relation R between two entities h (head) and t (tail). Moreover, we show that marginal loss minimisation, a popular objective used in much prior work in KGE, follows naturally from the loglikelihood ratio maximisation under the probabilities estimated from the KGEs according to our theoretical relationship. We propose a learning objective motivated by the theoretical analysis to learn KGEs from a given knowledge graph. Using the derived objective, accurate KGEs are learnt from FB15K237 and WN18RR benchmark datasets, providing empirical evidence in support of the theory. * Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the University of Liverpool and is not associated with Amazon.

show abstract

“…Note that c has a uniform distribution over the unit sphere. In this case, from Lemma A.5 in (Arora et al, 2015), (53) holds approximately.…”

Section: Learning With Multiple Negative Triplesmentioning

confidence: 89%

Section: B Proof Of Relwalk Theoremmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

RelWalk - A Latent Variable Model Approach to Knowledge Graph Embedding

Bollegala

Hakami

Yoshida

et al. 2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

View full text Add to dashboard Cite

show abstract

“…In this paper, we try to learn a model for word embeddings that incorporates syntactic information and naturally leads to better compositions for syntactically related word pairs. Our model is motivated by the principled approach for understanding word embeddings initiated by Arora et al (2015), and models for composition similar to Coecke et al (2010). Arora et al (2015) gave a generative model (RAND-WALK) for word embeddings, and showed several previous algorithms can be interpreted as finding the hidden parameters of this model.…”

Section: Introductionmentioning

confidence: 99%

“…Our model is motivated by the principled approach for understanding word embeddings initiated by Arora et al (2015), and models for composition similar to Coecke et al (2010). Arora et al (2015) gave a generative model (RAND-WALK) for word embeddings, and showed several previous algorithms can be interpreted as finding the hidden parameters of this model. However, the RAND-WALK model does not treat syntactically related word-pairs differently from other word pairs.…”

Section: Introductionmentioning

confidence: 99%

Understanding Composition of Word Embeddings via Tensor Decomposition

Frandsen

2019

Preprint

View full text Add to dashboard Cite

Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition -given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The result of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical results with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method.

show abstract

Learning Word Embeddings from the Portuguese Twitter Stream: A Study of Some Practical Aspects

Saleiro

Sarmento

Rodrigues

et al. 2017

Progress in Artificial Intelligence

View full text Add to dashboard Cite

This paper describes a preliminary study for producing and distributing a large-scale database of embeddings from the Portuguese Twitter stream. We start by experimenting with a relatively small sample and focusing on three challenges: volume of training data, vocabulary size and intrinsic evaluation metrics. Using a single GPU, we were able to scale up vocabulary size from 2048 words embedded and 500K training examples to 32768 words over 10M training examples while keeping a stable validation loss and approximately linear trend on training time per epoch. We also observed that using less than 50% of the available training examples for each vocabulary size might result in overfitting. Results on intrinsic evaluation show promising performance for a vocabulary size of 32768 words. Nevertheless, intrinsic evaluation metrics suffer from oversensitivity to their corresponding cosine similarity thresholds, indicating that a wider range of metrics need to be developed to track progress.

show abstract

A Latent Variable Model Approach to PMI-based Word Embeddings

Cited by 13 publications

References 0 publications

RelWalk - A Latent Variable Model Approach to Knowledge Graph Embedding

RelWalk - A Latent Variable Model Approach to Knowledge Graph Embedding

Understanding Composition of Word Embeddings via Tensor Decomposition

Learning Word Embeddings from the Portuguese Twitter Stream: A Study of Some Practical Aspects

Contact Info

Product

Resources

About