SSE-PT: Sequential Recommendation Via Personalized Transformer

Abstract

Temporal information is crucial for recommendation problems because user preferences are naturally dynamic in the real world. Recent advances in deep learning, especially the discovery of various attention mechanisms and newer architectures in addition to widely used RNN and CNN in natural language processing, have allowed for better use of the temporal ordering of items that each user has engaged with. In particular, the SASRec model, inspired by the popular Transformer model in natural languages processing, has achieved state-of-the-art results. However, SASRec, just like the original Transformer model, is inherently an un-personalized model and does not include personalized user embeddings. To overcome this limitation, we propose a Personalized Transformer (SSE-PT) model, outperforming SASRec by almost 5% in terms of NDCG@10 on 5 real-world datasets. Furthermore, after examining some random users' engagement history, we find our model not only more interpretable but also able to focus on recent engagement patterns for each user. Moreover, our SSE-PT model with a slight modification, which we call SSE-PT++, can handle extremely long sequences and outperform SASRec in ranking results with comparable training speed, striking a balance between performance and speed requirements. Our novel application of the Stochastic Shared Embeddings (SSE) regularization is essential to the success of personalization. Code and data are open-sourced at https://github.com/wuliwei9278/SSE-PT.

Context-Aware Recommender Systems for Social Networks: Review, Challenges and Opportunities

“…Table III shows the distribution of studies based on the different techniques that are used in recommendation approaches. Techniques References Bayesian probability [80], [90] Bipartite graphs embedding [84] Clustering [30], [60], [70], [91], [107] Decision tree [26], [38], [83] Deep neural networks [35], [56], [59], [65], [71], [72], [73], [76], [93], [98], [99], [100], [108] Distance-based method [88], [89] Fuzzy clustering [113] K Nearest Neighbor [24], [25], [44], [54], [75]…”

“…The neural network musical recommender system proposed in [108] aggregates the embeddings of music pieces in their complete listening records and active interaction session, respectively, to derive users' general and contextual preferences. Similarly, the recommender system proposed in [35] incorporated the embeddings of user and item in the NN model for better personalization to identify dynamic preferences based on user's interaction with items. The graph-attention neural network proposed in [99] relies on dynamic user's behaviors with recurrent neural network (RNN) and context-dependent social influence to model user's session-based interest and forceful social impacts.…”

Deep Self-Attention for Sequential Recommendation (S)

“…BERT4rec [7] employed the deep bidirectional self-attention to model user behavior sequences. SSE-PT [23] introduced additional personalized embeddings to improve the performance of Transformer for sequential recommendation. Although the previous methods have been proven effective, they ignore the limitation of dotproduct when the size of embedding vector is small.…”

“…Following previous methods [6], [23], we adopt a binary cross entropy loss to optimize DSASrec, which is defined as:…”

Contextual BERT: Conditioning the Language Model Using a Global State

“…Chen et al (2019) condition on a customer's variably-sized click history using a Transformer (Vaswani et al, 2017). The most similar to our work are Wu et al (2020) who personalize by concatenating every position in the input sequence with a user embedding -method [C] from Section 3. Their approach, however, lacks an architectural bias that makes the model treat the user embedding as global information.…”