Improving Negative Sampling for Word Representation using Self-embedded Features

Abstract

Although the word-popularity based negative sampler has shown superb performance in the skip-gram model, the theoretical motivation behind oversampling popular (non-observed) words as negative samples is still not well understood. In this paper, we start from an investigation of the gradient vanishing issue in the skip-gram model without a proper negative sampler. By performing an insightful analysis from the stochastic gradient descent (SGD) learning perspective, we demonstrate that, both theoretically and intuitively, negative samples with larger inner product scores are more informative than those with lower scores for the SGD learner in terms of both convergence rate and accuracy. Understanding this, we propose an alternative sampling algorithm that dynamically selects informative negative samples during each SGD update. More importantly, the proposed sampler accounts for multi-dimensional self-embedded features during the sampling process, which essentially makes it more effective than the original popularity-based (one-dimensional) sampler. Empirical experiments further verify our observations, and show that our fine-grained samplers gain significant improvement over the existing ones without increasing computational complexity.

Batch IS NOT Heavy: Learning Word Representations From All Samples

“…• SG: This is the original skip-gram model with SGD and negative sampling (Mikolov et al, 2013a,b). • SGA: This is the skip-gram model with an adaptive sampler (Chen et al, 2018). • GloVe: This method applies biased MF on the positive samples of word co-occurrence matrix (Pennington et al, 2014).…”

“…for each model are given from left to right as follows. SG: subsampling of frequent words, window size and the number of negative samples; SGA: λ (Chen et al, 2018) that controls the distribution of the rank, the other parameters are the same with SG; GloVe: xmax, window size and symmetric window; LexVec: subsampling of frequent words and the number of negative samples; AllVec: the negative weight α0 and δ. Boldface denotes the highest total accuracy. Figure 2(a) shows the impact of the overall weight α 0 by setting δ as 0.75 (inspired by the setting of skip-gram).…”

“…By far, most state-of-the-art embedding methods rely on SGD and negative sampling for optimization. However, the performance of SGD is highly sensitive to the sampling distribution and the number of negative samples (Chen et al, 2018;Yuan et al, 2016), as shown in Figure 1. Essentially, sampling is biased, making it difficult to converge to the same loss with all examples, regardless of how many update steps have been taken.…”

“…This suggests that the sampling distribution (of negative words) has a great impact on the embedding quality. Furthermore, Chen et al (2018) and Guo et al (2018) recently found that replacing the original sampler with adaptive samplers could result in better performance. The adaptive samplers are used to find more informative negative examples during the training process.…”

Relaxed softmax for PU learning

“…Contrary to conventional generic sampling losses such as Sampled Softmax or Noise Contrastive Estimation, we advocate that negative sampling schemes should not try to approximate the softmax function but rather act as a way to induce an informative bias on the model of negatives. To this end, authors in [7] provide an insightful analysis of negative sampling and show that negative samples that have high inner product scores with the context word provide more informative gradients. PU learning approaches have already shown some successes in the context of Recommender Systems as shown in [20].…”