Self-supervised Learning for Large-scale Item Recommendations

Yao, Tiansheng; Yi, Xinyang; Cheng, Derek Zhiyuan; Yu, Felix X.; Chen, Ting; Menon, Aditya Krishna; Hong, Lichan; H., Ed; Tjoa, Steven K.; Kang, Jieqi; Ettinger, Evan

doi:10.1145/3459637.3481952

Cited by 158 publications

(61 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the most typical dropout setup, we apply dropout only to the fully-connected layers [14,31]. In particular, we neither enable dropout on convolution layers on EMNIST, nor on embeddings [55].…”

Section: Variation Estimation Task Configurationsmentioning

confidence: 99%

Dropout Prediction Uncertainty Estimation Using Neuron Activation Strength

Yu¹,

Chen²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

It is well-known that deep neural networks generate different predictions even given the same model configuration and training dataset. It thus becomes more and more important to study prediction variation, the variation of the predictions on a given input example, in neural network models. Dropout has been commonly used in various applications to quantify prediction variations. However, using dropout in practice can be expensive as it requires running dropout inferences many times to estimate prediction variation.We study how to estimate dropout prediction variation in a resource-efficient manner. We demonstrate that we can use neuron activation strengths to estimate dropout prediction variation under different dropout settings and on a variety of tasks using three large datasets, MovieLens, Criteo, and EMNIST. Our approach provides an inference-once alternative to estimate dropout prediction variation as an auxiliary task. Moreover, we demonstrate that using activation features from a subset of the neural network layers can be sufficient to achieve variation estimation performance almost comparable to that of using activation features from all layers, thus reducing resources even further for variation estimation.

show abstract

Section: Variation Estimation Task Configurationsmentioning

confidence: 99%

Dropout Prediction Uncertainty Estimation Using Neuron Activation Strength

Yu¹,

Chen²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To characterize the intrinsic data correlations, S3Rec [44] combines SSL with sequential recommendation by utilizing four MIM objectives, i.e., Item-Attribute MIM, Sequence-Item MIM, Sequence-Attribute MIM, and Sequence-Sequence MIM. In large-scale item recommendations, an auxiliary SSL task is employed to explore feature correlations by applying different feature masking patterns [45]. CL4SRec [46] proposes three data augmentation techniques (i.e., cropping, masking and reordering) from which two methods are randomly sampled and applied to each user sequence.…”

Section: B Self-supervised Learningmentioning

confidence: 99%

“…As a result, similar interests can thus have similar representations (defined as alignment) and sufficient information are kept to distinguish different interests (defined as uniformity). 45 Aug the alignment and uniformity properties are necessary and important for a good SSL system, as proved in [52]. Formally, taking z i,1 p , z i,2 p from the same interest as positive pairs while z i,1 p , z i,2 p and z i,1 p , z i,2 p from different samples as negative pairs, the InfoNCE loss for learning the interest-level correlation is formulated as:…”

Section: B the Miss Frameworkmentioning

confidence: 99%

“…2) Superiority Analysis: To demonstrate the superiority of our proposed MISS framework, we compare it with state-ofthe-art self-supervised learning models. Specifically, we apply MISS, IRSSL [45], S3Rec [44], and CL4SRec [46] to the IPNN, DIN, and FiGNN models for embedding enhancement purpose. Besides above SSL models, we also equip these base CTR models with a rule based model that segments the behavior sequence into several sub-sequences based on item categories and then conduct dropout on each sequence for SSL.…”

Section: Ablation Studymentioning

confidence: 99%

See 1 more Smart Citation

MISS: Multi-Interest Self-Supervised Learning Framework for Click-Through Rate Prediction

Guo¹,

Zhang²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

CTR prediction is essential for modern recommender systems. Ranging from early factorization machines to deep learning based models in recent years, existing CTR methods focus on capturing useful feature interactions or mining important behavior patterns. Despite the effectiveness, we argue that these methods suffer from the risk of label sparsity (i.e., the user-item interactions are highly sparse with respect to the feature space), label noise (i.e., the collected user-item interactions are usually noisy), and the underuse of domain knowledge (i.e., the pairwise correlations between samples). To address these challenging problems, we propose a novel Multi-Interest Self-Supervised learning (MISS) framework which enhances the feature embeddings with interest-level self-supervision signals.With the help of two novel CNN-based multi-interest extractors, self-supervision signals are discovered with full considerations of different interest representations (point-wise and union-wise), interest dependencies (short-range and long-range), and interest correlations (inter-item and intra-item). Based on that, contrastive learning losses are further applied to the augmented views of interest representations, which effectively improves the feature representation learning. Furthermore, our proposed MISS framework can be used as an "plug-in" component with existing CTR prediction models and further boost their performances. Extensive experiments on three large-scale datasets show that MISS significantly outperforms the state-of-the-art models, by up to 13.55% in AUC, and also enjoys good compatibility with representative deep CTR models.

show abstract

“…Secondly, existing GNN based CF approaches rely on explicit interaction links for learning node representations, while high-order relations or constraints (e.g., user or item similarity) cannot be explicitly utilized for enriching the graph information, which has been shown essentially useful in recommendation tasks [24,27,35]. Although several recent studies leverage constative learning to alleviate the sparsity of interaction data [33,39], they construct the contrastive pairs by randomly sampling nodes or corrupting subgraphs. It lacks consideration on how to construct more meaningful contrastive learning tasks tailored for the recommendation task [24,27,35].…”

Section: Introductionmentioning

confidence: 99%

Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning

Lin,

Tian,

Hou

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, graph collaborative filtering methods have been proposed as an effective recommendation approach, which can capture users' preference over items by modeling the user-item interaction graphs. Despite the effectiveness, these methods suffer from data sparsity in real scenarios. In order to reduce the influence of data sparsity, contrastive learning is adopted in graph collaborative filtering for enhancing the performance. However, these methods typically construct the contrastive pairs by random sampling, which neglect the neighboring relations among users (or items) and fail to fully exploit the potential of contrastive learning for recommendation.To tackle the above issue, we propose a novel contrastive learning approach, named Neighborhood-enriched Contrastive Learning, named NCL, which explicitly incorporates the potential neighbors into contrastive pairs. Specifically, we introduce the neighbors of a user (or an item) from graph structure and semantic space respectively. For the structural neighbors on the interaction graph, we develop a novel structure-contrastive objective that regards users (or items) and their structural neighbors as positive contrastive pairs. In implementation, the representations of users (or items) and neighbors correspond to the outputs of different GNN layers. Furthermore, to excavate the potential neighbor relation in semantic space, we assume that users with similar representations are within the semantic neighborhood, and incorporate these semantic neighbors into the prototype-contrastive objective. The proposed NCL can be optimized with EM algorithm and generalized to apply to graph collaborative filtering methods. Extensive experiments on five public datasets demonstrate the effectiveness of the proposed NCL, notably with 26% and 17% performance gain over a competitive graph collaborative filtering base model on the Yelp and Amazon-book datasets, respectively. Our implementation code is available at: https://github.com/RUCAIBox/NCL. CCS CONCEPTS• Information systems → Recommender systems. † Equal contribution.

show abstract

Self-supervised Learning for Large-scale Item Recommendations

Cited by 158 publications

References 23 publications

Dropout Prediction Uncertainty Estimation Using Neuron Activation Strength

Dropout Prediction Uncertainty Estimation Using Neuron Activation Strength

MISS: Multi-Interest Self-Supervised Learning Framework for Click-Through Rate Prediction

Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning

Contact Info

Product

Resources

About