Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems

Abstract

Recommender System (RS) is an important online application that affects billions of users every day. The mainstream RS ranking framework is composed of two parts: a Multi-Task Learning model (MTL) that predicts various user feedback, i.e., clicks, likes, sharings, and a Multi-Task Fusion model (MTF) that combines the multi-task outputs into one final ranking score with respect to user satisfaction. There has not been much research on the fusion model while it has great impact on the final recommendation as the last crucial process of the ranking. To optimize long-term user satisfaction rather than obtain instant returns greedily, we formulate MTF task as Markov Decision Process (MDP) within a recommendation session and propose a Batch Reinforcement Learning (RL) based Multi-Task Fusion framework (BatchRL-MTF) that includes a Batch RL framework and an online exploration. The former exploits Batch RL to learn an optimal recommendation policy from the fixed batch data offline for long-term user satisfaction, while the latter explores potential highvalue actions online to break through the local optimal dilemma. With a comprehensive investigation on user behaviors, we model the user satisfaction reward with subtle heuristics from two aspects of user stickiness and user activeness. Finally, we conduct extensive experiments on a billion-sample level real-world dataset to show the effectiveness of our model. We propose a conservative offline policy estimator (Conservative-OPEstimator) to test our model offline. Furthermore, we take online experiments in a real recommendation environment to compare performance of different models. As one of few Batch RL researches applied in MTF task successfully, our model has also been deployed on a large-scale industrial short video platform, serving hundreds of millions of users.

Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation

“…In the future, when fitting user interests is not a bottleneck anymore, researchers could consider higher-level goals, such as pursuing users' long-term satisfaction [57] or optimizing social utility [17]. With the increase in high-quality offline data, we believe that offline RL can be better adapted to recommender systems to achieve these goals.…”

LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting

“…However, for a fair comparison to LabelCraft, we merge all explicit feedback as a label by computing 𝛿 ( (𝒚 𝑒 )) in Equation ( 9). • PC [38]. In this method, the watch time of a video is compared to the video duration to determine whether a user has fully watched the video, forming the Play Completion label.…”

Multi-Task Deep Recommender Systems: A Survey

“…By setting user-item features as state and continuous score pairs for multiple tasks as actions, the RL-based MTL method is capable of handling the sequential user-item interaction and optimizing long-term user engagement. Zhang et al [79] formulate MTF as Markov Decision Process and use batch Reinforcement Learning to optimize long-term user satisfaction. Han et al [22] propose to use an actor-critic model to learn the optimal fusion weight of CTR and the bid rather than greedy ranking strategies to maximize the long-term revenue.…”

“…On the other hand, social media is a more complex field since the users interact with both items and users. Multiple MTDRS models validate their effectiveness on social media by online A/B test, including MMoE [84] on YouTube considering engagement and satisfaction, LT4REC [69] on Tencent Video, and BatchRL-MTF [79] on Tencent short video platform.…”

“…Several works try to search for fusion weight, such as Grid Search [20,50], Evolutionary Algorithm [49], and Bayesian Optimization [16]. A state-of-the-art solution is RL [22,79], as discussed in Section 3.3.3.…”