Batch IS NOT Heavy: Learning Word Representations From All Samples

Abstract

Stochastic Gradient Descent (SGD) with negative sampling is the most prevalent approach to learn word representations. However, it is known that sampling methods are biased especially when the sampling distribution deviates from the true data distribution. Besides, SGD suffers from dramatic fluctuation due to the onesample learning scheme. In this work, we propose AllVec that uses batch gradient learning to generate word representations from all training samples. Remarkably, the time complexity of AllVec remains at the same level as SGD, being determined by the number of positive samples rather than all samples. We evaluate AllVec on several benchmark tasks. Experiments show that AllVec outperforms samplingbased SGD methods with comparable efficiency, especially for small training corpora.

Efficient Heterogeneous Collaborative Filtering without Negative Sampling for Recommendation

“…For example, in Table 2, the performances of ExpoMF are better than BPR; and our EHCF outperforms all the baselines. This is consistent with previous work (Yuan et al 2018;Xin et al 2018), which indicates that regardless of what sampler is utilized or how many updates are taken, sampling is still a biased approach. 3) Our EHCF significantly outperforms the state-of-the-art CF methods in both traditional (single-behavior) and heterogeneous scenarios.…”

“…While previous studies (He et al 2017;Wang et al 2018;Chen et al 2019b) have shown that the performance of NS is not robust as it is highly sensitive to the sampling distribution and the number of negative samples. Essentially, sampling is biased, making it difficult to converge to the optimal ranking performance regardless of how many update steps have been taken (Xin et al 2018). Besides, to leverage heterogeneous user behavior, NS strategy needs to sample a negative instance for every observed interaction (regardless of the behavior type), which produces a very large randomness in total (about K times than singlebehavior scenario where K is the number of behavior types).…”

Efficient Neural Matrix Factorization without Sampling for Recommendation

“…For example, in Table 4, the performances of WMF and ExpoMF are better than BPR; and our ENMF outperforms BPR, GMF, NCF and Con-vNCF. This is consistent with previous work [4,57,60], which indicates that regardless of what sampler is utilized or how many updates are taken, sampling is still a biased approach. (3) The results show that neural methods generally performs better than traditional collaborative filtering methods, which verifies the advantages of neural networks over traditional models in representation learning.…”

“…By negative sampling, the number of negative instances is greatly reduced, therefore the overall time complexity is controllable [23]. However, the downside is that sampling-based methods usually have a slower convergence rate and the performance is highly dependent of the design of the sampler [25,57]. Whole-data-based strategy [11,26,30,31] sees all the missing data as negative.…”

“…Another reasonable intuition is that the missing entries of active users (who have clicked many items) are more likely to be true negatives [1]. A commonly used frequency-based weighting strategy [25,57,60] is as follows:…”