Modeling Attrition in Recommender Systems with Departing Bandits

Abstract

Traditionally, when recommender systems are formalized as multi-armed bandits, the policy of the recommender system influences the rewards accrued, but not the length of interaction. However, in real-world systems, dissatisfied users may depart (and never come back). In this work, we propose a novel multi-armed bandit setup that captures such policy-dependent horizons. Our setup consists of a finite set of user types, and multiple arms with Bernoulli payoffs. Each (user type, arm) tuple corresponds to an (unknown) reward probability. Each user's type is initially unknown and can only be inferred through their response to recommendations. Moreover, if a user is dissatisfied with their recommendation, they might depart the system. We first address the case where all users share the same type, demonstrating that a recent UCB-based algorithm is optimal. We then move forward to the more challenging case, where users are divided among two types. While naive approaches cannot handle this setting, we provide an efficient learning algorithm that achieves O(sqrt(T)ln(T)) regret, where T is the number of users.

Learning with Exposure Constraints in Recommendation Systems

“…From a broader perspective, our work relates to bandits with complex rewards schemes, i.e., abandonment elements [3,11,44], and non-stationary rewards [4,22,26,27,34,38]. Our work is also related to multi-stakeholder recommendation systems [5,6,9,30,39,45] and fairness in machine learning [12,29,43].…”

“…Notice that by the end of the exploration stage, all arms are still viable. To see this, recall that our assumption in Inequality (3) suggests that at some point in every phase, all arms a will be pulled at least max(δ a , γτ ) times, which is by definition greater than the exposure constraint δ a .…”

“…We further denote by x u,a the number of rounds that a ∈ K is pulled for type u-users during the exploration stage. The arm a is pulled at least γ • τ rounds in a phase (see Inequality (3) and Lines 3-4 in Algorithm 3) and thus, at least T 2 /3 times during the exploration stage. In addition, in the exploration stage the pulled arm a t is independent of the arrival type u t ; therefore, x u,a ∼ bin(N, P u ), for N ≥ T 2 /3 .…”

“…However, due to their rapid adoption in commercial applications, many RSs are now dynamic economic systems with multiple stakeholders, facing challenges beyond dissolving uncertainty in matching. Fairness [6,15,18,33], misinformation [17], user incentives [3,24], and privacy [21] are only some of the challenges RSs face.…”

Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective

“…Several papers have proposed multi-armed bandit models where surrogate outcomes encode actions' longterm impacts. These include bandit models where poor recommendations cause attrition [Ben-Porat et al, 2022, Bastani et al, 2022 and bandit models where objectives incorporate diversity/boredom considerations [Xie et al, 2022, Cao et al, 2020, Ma et al, 2016. Wu et al [2017] studies a variation on typical bandit model where actions impact whether a user will return to the system.…”