Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions

Abstract

Users of music streaming, video streaming, news recommendation, and e-commerce services often engage with content in a sequential manner. Providing and evaluating good sequences of recommendations is therefore a central problem for these services. Prior reweighting-based counterfactual evaluation methods either suffer from high variance or make strong independence assumptions about rewards. We propose a new counterfactual estimator that allows for sequential interactions in the rewards with lower variance in an asymptotically unbiased manner. Our method uses graphical assumptions about the causal relationships of the slate to reweight the rewards in the logging policy in a way that approximates the expected sum of rewards under the target policy. Extensive experiments in simulation and on a live recommender system show that our approach outperforms existing methods in terms of bias and data e ciency for the sequential track recommendations problem.

On Skipping Behaviour Types in Music Streaming Sessions

“…For a fair comparison they are therefore ignored when answering RQ2. Time of The Day: We define 5 time windows, where numbers in brackets correspond to the hour range: night (0-5), morning (6)(7)(8)(9)(10)(11), afternoon (12)(13)(14)(15)(16)(17), evening (18)(19)(20)(21)(22)(23), all (0-23). If a session spans across two hours, we round up and consider the whole session as either part of start or end hour.…”

“…Modelling and understanding skipping behaviour in music listening sessions arguably plays a crucial role in understanding user behaviour in modern streaming services. For instance, the skipping signal has already been used as a measure in heuristic-based playlist generation systems [9,25], user satisfaction [16,28], relevance [17], and as counterfactual estimators in Recommender Systems [22].…”

Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model

“…One way to reduce the variance is introducing a reasonable assumption on user behavior to make the combinatorial item space tractable. However, unrealistically strong assumptions may cause serious bias in OPE [13]. Therefore, achieving a well-balanced bias-variance tradeoff by introducing an appropriate user behavior assumption is the key for enabling accurate OPE of ranking policies.…”

“…setting, however, IPS can suffer from large variance, as the item space is combinatorially large [12,13,23]. In contrast, Independent IPS (IIPS) is based on the independence assumption to address the variance issue [12].…”

“…To address the variance issue of IPS and the bias issue of IIPS, McInerney et al [13] proposed Reward interaction IPS (RIPS), which relies on the cascade assumption. The cascade assumption assumes that a user interacts with a list of items sequentially from the top position to the bottom [6].…”