Word Embedding as Maximum A Posteriori Estimation

Abstract

The GloVe word embedding model relies on solving a global optimization problem, which can be reformulated as a maximum likelihood estimation problem. In this paper, we propose to generalize this approach to word embedding by considering parametrized variants of the GloVe model and incorporating priors on these parameters. To demonstrate the usefulness of this approach, we consider a word embedding model in which each context word is associated with a corresponding variance, intuitively encoding how informative it is. Using our framework, we can then learn these variances together with the resulting word vectors in a unified way. We experimentally show that the resulting word embedding models outperform GloVe, as well as many popular alternatives.

Word and Document Embedding with vMF-Mixture Priors on Context Word Vectors

“…In this section, we describe the word embedding results, where we directly compare our model with the following baselines: GloVe (Pennington et al, 2014), Skipgram (Mikolov et al, 2013b) (denoted as SG), Continuous Bag of Words (Mikolov et al, 2013b) (denoted as CBOW), and the recently proposed WeMAP model (Jameel et al, 2019). We have used the Wikipedia dataset which was shared by Jameel et al (2019), using the same vocabulary and preprocessing strategy. We report results for 300-dimensional word vectors and we use K = 3000 mixture components for our model.…”

“…(sHDP) 1314 , 7) GloVe 15 (Pennington et al, 2014), 8) WeMAP (Jameel et al, 2019), 9) Skipgram (SG) and Continuous Bag-of-Words 16 (Mikolov et al, 2013b) models. In the case of the word embedding models, we create document vectors in the same way as we do for our model, by simply replacing the role of target word vectors with document word vectors.…”

“…The advantage of this probabilistic formulation is that it allows us to introduce priors on the parameters of the model. This strategy was recently used in the WeMAP model (Jameel et al, 2019) to replace the constant variance σ 2 by a variance σ 2 j that depends on the context word. In this paper, however, we will use priors on the parameters of the word embedding model itself.…”

“…, where we have considered the same datasets asJameel et al (2019).In the table, we refer to EN-RW-Stanford as Stanf, EN-SIMLEX-999 as LEX, SimVerb3500 as Verb, EN-MTurk771 as Tr771, EN-MTurk287 as Tr287, EN-MENTR3K as TR3k, the RareWords dataset as RW, and the recently introduced Card-660 rare words dataset (Pilehvar et al, 2018) denoted as CA-660. Note that we have removed multi-word expressions from the RW-660 dataset and consider only unigrams, which reduces the size of 62.81 53.04 55.21 14.82 10.56 0.881 SG 71.58 60.50 51.71 55.45 13.48 08.78 0.671 CBOW 64.81 47.39 45.33 50.58 10.11 07.02 0.764 WeMAP 83.52 63.08 55.08 56.03 14.95 10.62 0.903 CvMF 63.22 67.41 63.21 65.94 17.46 9.380 1.100 CvMF(NIG) 64.14 67.55 63.55 65.95 17.49 9.410 1.210 Word analogy accuracy results on different datasets.…”

MAP Estimation With Bernoulli Randomness, and Its Application to Text Analysis and Recommender Systems

“…It plays an essential role in various practical scenarios where there exist hidden variables or uncertainty. Some applications include image processing [3], [4], text analysis [5]- [7], recommender system [8], protein design and protein side-chain prediction problems [9], [10]. Adding the prior probability information reduces the overdependence on the observed data for parameter estimation, MAP estimation be seen as a regularization of Maximum Likelihood Estimation (MLE), MAP can deal well with low training data.…”