On the Effectiveness of Sampled Softmax Loss for Item Recommendation

Abstract

Learning objectives of recommender models remain largely unexplored. Most methods routinely adopt either pointwise (e.g., binary cross-entropy) or pairwise (e.g., BPR) loss to train the model parameters, while rarely pay attention to softmax loss due to the high computational cost. Sampled softmax loss emerges as an efficient substitute for softmax loss. Its special case, InfoNCE loss, has been widely used in self-supervised learning and exhibited remarkable performance for contrastive learning. Nonetheless, limited studies use sampled softmax loss as the learning objective to train the recommender. Worse still, none of them explore its properties and answer "Does sampled softmax loss suit for item recommendation?" and "What are the conceptual advantages of sampled softmax loss, as compared with the prevalent losses?", to the best of our knowledge.In this work, we aim to better understand sampled softmax loss for item recommendation. Specifically, we first theoretically reveal three model-agnostic advantages: (1) mitigating popularity bias, which is beneficial to long-tail recommendation; (2) mining hard negative samples, which offers informative gradients to optimize model parameters; and (3) maximizing the ranking metric, which facilitates top-𝐾 performance. Moreover, we probe the model-specific characteristics on the top of various recommenders. Experimental results suggest that sampled softmax loss is more friendly to historyand graph-based recommenders (e.g., SVD++ [21] and LightGCN [13]), but performs poorly for ID-based models (e.g., MF [22]). We ascribe this to its shortcoming in learning representation magnitude, making the combination with the models that are also incapable of adjusting representation magnitude learn poor representations. In contrast, the history-and graph-based models, which naturally adjust representation magnitude according to node degree, are able to compensate for the shortcoming of sampled softmax loss. We will release our implementations upon acceptance.

Rethinking Multi-Interest Learning for Candidate Matching in Recommender Systems

“…In prior works [2][3][4]50], 𝑄 is often generic sampling distributions such as log-uniform and uniform sampling, which are shown to perform relatively well in general RS learning [44]. Specifically, using log-uniform sampling on the item set sorted by popularity gives the popular items a higher probability of being selected as negative samples.…”

“…First, we raise doubts about the effectiveness of uniformly sampled softmax in multi-interest scenarios. While uniformly sampled softmax has been shown to be effective in training general recommendation systems [44], it falls short in multi-interest recommendation systems, such as ComiRec [2] and PIMIRec [4]. As illustrated in Figure 1 (a), it has significantly worse performance compared to full softmax within a reasonable sample size range (e.g., below a thousand).…”

Adap-$τ$: Adaptively Modulating Embedding Magnitude for Recommendation

“…There are multiple choices of loss functions for training a recommendation model including pointwise loss (e.g., BCE [12,26], MSE [9,17]), pairwise loss(e.g., BPR [25]) and Softmax loss [34]. Recent work [34] finds Softmax loss could mitigate popularity bias, achieves great training stability, and aligns well with the ranking metric. It usually achieves better performance than others and thus attracts a surge of interest in recommendation.…”

“…1 a). Here we follow [34] and split items into ten groups in terms of item popularity. The larger group ID indicates the group contains more popular items.…”

“…Temperature has exhibited its capability in numerous fields such as CV and NLP in particular with contrastive learning [3,8]. Motivated by the success in other areas, recommendation combined with contrastive learning has received scant attention in recent research literature [21,28,33,34,38]. Although normalizaiton and 𝜏 are heuristically used by a small amount of work, it still lacks comprehensive exploration in recommendation.…”