WordRank: Learning Word Embeddings via Robust Ranking

Abstract

Embedding words in a vector space has gained a lot of attention in recent years. While stateof-the-art methods provide efficient computation of word similarities via a low-dimensional matrix embedding, their motivation is often left unclear. In this paper, we argue that word embedding can be naturally viewed as a ranking problem due to the ranking nature of the evaluation metrics. Then, based on this insight, we propose a novel framework WordRank that efficiently estimates word representations via robust ranking, in which the attention mechanism and robustness to noise are readily achieved via the DCG-like ranking losses. The performance of WordRank is measured in word similarity and word analogy benchmarks, and the results are compared to the state-of-the-art word embedding techniques. Our algorithm is very competitive to the state-of-the-arts on large corpora, while outperforms them by a significant margin when the training set is limited (i.e., sparse and noisy). With 17 million tokens, WordRank performs almost as well as existing methods using 7.2 billion tokens on a popular word similarity benchmark. Our multi-node distributed implementation of WordRank is publicly available for general usage.

Approximating Word Ranking and Negative Sampling for Word Embedding

“…• CBOW-a [Chen et al, 2017]: a CBOW variant which adaptively sample negative words by ranking scores. • WordRank [Ji et al, 2015]: a ranking model that puts more weights on positive words by rank values. • OptRank: our ranking model with the optimization in both positive word ranking and negative word sampling.…”

“…We set ε as 0.5 in five subsets and 1.0 in Wiki2017(14G). For the WordRank model, we adopt the settings given by [Ji et al, 2015]: logarithm as the objective function, initial value of scale parameter is α = 100 and offset parameter β = 99. The dimension of word vectors is also set to 300.…”

“…An adaptive sampler [Chen et al, 2017] has been proposed to roughly select the negative words which have larger inner products with contextual words than positive words, but it does not take care of the issue of positive word weighing. The WordRank model [Ji et al, 2015] proposes to treat the word embedding as a ranking problem. The similarity is computed between the contextual words and a positive word and then fed into a ranking function, the result of which is adopted as the weights of positive words.…”

Word-Embedding Benchmarking

“…Das aktuellste Word-Embedding mit Namen BERT (1032 Zitationen), ist zum Zeitpunkt der Erstellung des Beitrages von keiner frei zugänglichen Bibliothek bereitgestellt worden und ist demnach nicht für ein Benchmarking geeignet. [10], [13], [16], [17], [18], [19], [20], [5], [6], [14] Abbildung…”

Improving Negative Sampling for Word Representation using Self-embedded Features

“…In this subsection, we provide an intuitive example to explain the merits of popularity oversampling from ranking perspective. The reason is that training word embedding can also be naturally viewed as a ranking task that ranks an observed context word c P higher than any non-observed context word c N [14]. To illustrate this, we give a schematic of a ranked list for a target word w as below, where +1 and -1 denote an observed and non-observed context word respectively.…”