Modeling Attrition in Recommender Systems with Departing Bandits

Abstract

Traditionally, when recommender systems are formalized as multi-armed bandits, the policy of the recommender system influences the rewards accrued, but not the length of interaction. However, in real-world systems, dissatisfied users may depart (and never come back). In this work, we propose a novel multi-armed bandit setup that captures such policydependent horizons. Our setup consists of a finite set of user types, and multiple arms with Bernoulli payoffs. Each (user type, arm) tuple corresponds to an (unknown) reward probability. Each user's type is initially unknown and can only be inferred through their response to recommendations. Moreover, if a user is dissatisfied with their recommendation, they might depart the system. We first address the case where all users share the same type, demonstrating that a recent UCBbased algorithm is optimal. We then move forward to the more challenging case, where users are divided among two types. While naive approaches cannot handle this setting, we provide an efficient learning algorithm that achieves Õ( √ T ) regret, where T is the number of users.

Modeling Attrition in Recommender Systems with Departing Bandits

“…These results suggest an algorithm that achieves Õ( √ T ) regret in this setting. In the full version of this paper (Ben-Porat et al 2022), we also show an efficient optimal planning algorithm for multiple user types and two recommendation categories, and describe a scheme to construct semi-synthetic problem instances for this setting using real-world datasets.…”

“…These lemmas allow us to use Algorithm 2 with a policy set Π that contains all the fixed-arm policies, and derive a Õ( √ T ) regret bound. All omitted proofs can be found in the full version of this paper (Ben-Porat et al 2022).…”

Exploration, Exploitation, and Engagement in Multi-Armed Bandits with Abandonment

“…In contrast, in our MAB-A setting, the reward is unknown and a higher reward makes the user less likely to abandon the system. The concept of abandonment also appears in the sequential choice bandit problem in [4] and the departing bandit problem in [2]. However, the abandonment probabilities in their models do not depend on the past experience of the user.…”

“…Hence by Theorem 10 in [7], we have P `µpa i q ă μ0 t pa i q ˘ď e rrlog t `4 logplog tqs log ts tplog tq 4 ď 6e tplog tq 2 (91) for any t ě T 1 . Hence, we have…”

“…Bernoulli rewards of pulling arm a i , and the last inequality is due to the fact that tA t " a i , N t pa i q " nu 8 t"n`1 are mutually exclusive and the countable additivity of probability measure. Then by Hoeffding inequality and (108), we have 8 ÿ t"M `1 P `At " a i , N t pa i q ě Bptq, μ1 t pa i q ě µpa 1 q ď 8 ÿ n"1 exp ´´2n pr pa i q ´µpa i qq 2 ď 1 2 pr pa i q ´µpa i qq 2 (109)…”