Top-K Off-Policy Correction for a REINFORCE Recommender System

Abstract

Industrial recommender systems deal with extremely large action spaces -many millions of items to recommend. Moreover, they need to serve billions of users, who are unique at any point in time, making a complex user state space. Luckily, huge quantities of logged implicit feedback (e.g., user clicks, dwell time) are available for learning. Learning from the logged feedback is however subject to biases caused by only observing feedback on recommendations selected by the previous versions of the recommender. In this work, we present a general recipe of addressing such biases in a production top-K recommender system at YouTube, built with a policy-gradient-based algorithm, i.e. REINFORCE [48]. The contributions of the paper are: (1) scaling REINFORCE to a production recommender system with an action space on the orders of millions;(2) applying off-policy correction to address data biases in learning from logged feedback collected from multiple behavior policies; (3) proposing a novel top-K off-policy correction to account for our policy recommending multiple items at a time; (4) showcasing the value of exploration. We demonstrate the efficacy of our approaches through a series of simulations and multiple live experiments on YouTube.

Fairness in Recommendation Ranking through Pairwise Comparisons

“…Building real-world recommenders face a variety of challenges. Two that relate to the challenges in fairness are the temporal dynamics [33,48,26,9] and biased training data [29,15,3]. These issues do not just make training difficult but also evaluation of recommender performance [42].…”

“…We consider a production recommender system that is recommending a personalized list of K items to users. We consider a cascading recommender [47,24,16], with a set of retrieval systems [15] followed by a ranking system [16,36]. We assume that the retrieval systems return a set R of M relevant items from the total corpus J of M items, where M M ≥ K. The ranking model must then score and rank M items in R to get a final ranking of K items.…”

Estimating Attention Flow in Online Video Networks

“…Other enhancements include incorporating contextual data [5]. Most recently, Chen et al [10] and Ie et al [23] showed success in applying reinforcement learning techniques in YouTube recommender systems. Our work does not deal with designing a recommender system, nor does it attempt to reverse engineer the YouTube recommender.…”

“…Contrasting the extensive literature on evaluating the accuracy of recommendation [5,10,30,54], we focus on prior work that connects network structure with content consumption. Carmi et al [8] reported how the book sales on Amazon react to exogenous demand shocks -not only did the sales increase for the featured item, but the increase also propagated a few hops away by following the links created by the recommender systems.…”

“…The first gap measures and estimates the effects of recommender systems in complex social systems. The main goals of recommender systems are maximizing the chance that a user clicks on an item in the next step [4,16,17,48] or in a longer time horizon [5,10,23]. However, recommendation in social systems remains as an open problem for two reasons: (1) a limited conceptual understanding of how finite human attention is allocated over the network of content, in which some items gain popularity at the expense of, or with the assistance of others; (2) the computational challenge of jointly recommending a large collection of items.…”